Jeff Dean: A 10x Improvement Is the Signal of a Foundational Idea — MoE Proved It
dawnsongtweets · x · 2026-09-21
Part 3 of the Jeff Dean conversation series: how to recognize a foundational idea early.
- With Mixture-of-Experts, Dean recalled seeing 10x better training-compute-to-quality ratios than dense models — that was the signal: "When you see 10x improvements… it's probably going to be an important idea."
- Context: his Discovery Loop work on recursive self-improvement — using ML to improve ML (architecture search, Evolved Transformer were early examples), now at the scope of automating the entire AI improvement loop: architecture, data, evals, experiments, feedback
- He poses the question: what is today's 10x idea?
More from AGI Musings
- HF researcher asks: where do agents pick up their human-like lingo, and is there research on collab mixtures? — mervenoyann · 2026-09-22
- Citizens Advice urges 'right to talk to a human' as half of users find service chatbots unhelpful — nordicinst · 2026-09-22
- Every major AI company is converging on the same personal agent stack, with little differentiation — signulll · 2026-09-22
- NYT Essay: AI Slopware Is Everywhere Now — and Nobody Is Using It — round · 2026-09-22
- What happens to Amazon's ad business when AI agents do the comparison shopping? — firstadopter · 2026-09-22
- Dev's Reversal Take: Tickets Will Use Agents to Find Buyers, Not the Other Way — cocktailpeanut · 2026-09-22