Jeff Dean: A 10x Improvement Is the Signal of a Foundational Idea — MoE Proved It
dawnsongtweets · x · 2026-09-21
Part 3 of the Jeff Dean conversation series: how to recognize a foundational idea early.
- With Mixture-of-Experts, Dean recalled seeing 10x better training-compute-to-quality ratios than dense models — that was the signal: "When you see 10x improvements… it's probably going to be an important idea."
- Context: his Discovery Loop work on recursive self-improvement — using ML to improve ML (architecture search, Evolved Transformer were early examples), now at the scope of automating the entire AI improvement loop: architecture, data, evals, experiments, feedback
- He poses the question: what is today's 10x idea?
More from AGI Musings
- Why Is Inkling 2 Small Trained on So Few Tokens Yet Still Competitive? — georgejrjrjr · 2026-11-12
- Researcher willing to bet $100K the intelligence explosion won't happen next year — NathanpmYoung · 2026-11-12
- ICLR reviewer: the worst parts of my job are reading and answering AI slop — chhaviyadav_ · 2026-11-12
- Agent Swarms Are Autonomously Hacking Critical Infrastructure Run by Underpaid Devs — menhguin · 2026-11-12
- Fermi paradox twist: advanced civilizations may go digital after developing AI — VraserX · 2026-11-12
- Mark Cuban: AI could make healthcare costs 'drop like a rock' — alvelda · 2026-11-12