Andrew Dai: pre-training + fine-tuning was born from a forgotten checkpoint bug
ziv_ravid · x · 2026-09-27
- New episode of The Information Bottleneck features Andrew Dai, co-founder and CEO of Elorian AI, on vision models. Dai spent 14 years at Google.
- Dai co-authored the 2015 paper introducing pre-training and fine-tuning for language models — a result he says came from a bug: he forgot to update a checkpoint directory and accidentally fine-tuned a language model.
- The episode also covers why Google fell behind early in the LLM race, and why models still can't count objects in a photo. Dai argues much reasoning is visual, which is what Elorian is building for.
More from Fun
- 900+ to attend 'pro-data center party' in D.C. with AI cocktails and GenAI photo booth — Polymarket · 2026-09-27
- Stable Audio 3's "imperfections" defended: hallucination is the charm of AI music — Merzmensch · 2026-09-27
- Critics mock Gary Marcus for dodging past predictions with "it's not a pure LLM anymore" — inductionheads · 2026-09-27
- GPT-6 Astra beats national-level Yu-Gi-Oh player using browser control — MikePFrank · 2026-09-27
- Opus proposes planting bugs in its own code to test AI reviewers — Aizkmusic · 2026-09-27
- Thiel's 'flying cars' and '140 characters' may soon be the same guy — seanmcdonaldxyz · 2026-09-27