Just finished my internship at a fintech company (think of Robinhood) and learned something they definitely don't teach you in school.
So I was on the data platform team and there was this whole drama while I was there. Basically the company went all-in on one of the big cloud vendors for their data stuff a couple years ago. Everything was integrated, their catalog, governance, the whole stack. Then out of nowhere the vendor raised prices like 40% with barely any notice. The engineering managers were losing their minds.
The problem? All the company's metadata was trapped in the vendor's proprietary system. Like literally years of data lineage, access policies, everything. They couldn't just switch because migrating would take almost a year of engineering time. So they just had to pay lol.
One of the senior engineers told me this happens all the time. He called it "the walled garden trap." Big companies like Databricks, Snowflake, AWS all want you locked into their ecosystem. Once you're in, switching costs are insane.
Made me start reading about open source alternatives. Found this article that explains it way better than I can: https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e](https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e
There's this Apache project called Gravitino that's basically an open source metadata layer so you're not dependent on one vendor. Kinda interesting from a systems design perspective tbh.
Anyway this whole experience made me think:
Do companies actually consider vendor lock-in when choosing tech stacks or do they just go with whatever's easiest?
For those who've worked at startups vs big tech, is this more of a startup problem?
Is open source infra actually viable for most companies or is it too much ops overhead?
Anyone else learn stuff like this during internships that completely changed how you think about tech decisions?
Feels like we learn algorithms and data structures but nobody talks about the actual business/infrastructure tradeoffs you deal with in industry.