Engineer says a scratch-built data platform now serves 400M users internally
teodorio · x · 2026-07-25
An engineer says they built a data platform from scratch for 400 million users and it has grown into something used by tens of internal teams.
The stack now includes resilient in-session identity resolution, custom sources, about 500 dbt models, a metric store, event and metric definition APIs, CI/CD, QA environments, agent-based backfills, and an automated data-quality agent. The post frames it as a personal milestone after starting from ad hoc Airflow queries, notebooks, and petabytes of third-party analytics data.
More from Companies & People
- Prentis, New AI Lab by Reid Hoffman and Marc Pincus, Seeks $100M — TechCrunch AI · 2026-07-25
- A16z survey names Elad Gil, Marc Andreessen and Elon Musk as top angels — jfischoff · 2026-07-25
- The Washington Post runs Ask The Post AI on independent inference with 1.79B tokens a month — togethercompute · 2026-07-25
- Opus 5 powers a live AI tutor prototype for AlphaSchool — mattshumer_ · 2026-07-25
- VaynerX says a $20,000 internal app was prototyped in hours with Claude — tomcrawshaw01 · 2026-07-25
- ECCV Faces Backlash Over Forced Full Registrations for Visa-Blocked Students — gan_chuang · 2026-07-25