OpenAI researcher: intuitions from tiny models are useless for GPT-6-scale training
willdepue · x · 2026-11-06
OpenAI's willdepue makes his core argument in the double descent debate: intuitions derived from tiny, long-ago experiments are generally useless when reasoning about training models the size of GPT-6.
The remark directly counters opponents citing 2019-era 5-layer ResNet experiments to extrapolate scaling behavior — implying small-scale findings like double descent can't guide decisions about frontier-scale training.
Related event: OpenAI Researcher Debates Data Wall and Double Descent Theory on X(7 posts)→
More from AGI Musings
- "Reverse Alignment" initiative argues society—not just models—needs to be aligned for AI — profjamesevans · 2026-11-06
- Evans: pay data creators by contribution and surprise, verify via webs of attestation — profjamesevans · 2026-11-06
- James Evans & Glen Weyl in Science: institutions, not models, now bind AI's societal value — profjamesevans · 2026-11-06
- Greg Brockman says AI could make death optional, with disease-curing breakthroughs in 12-18 months — kimmonismus · 2026-11-06
- UK Survey: Most Think AI Will Take Jobs – Just Not Theirs — HotMinuteUK · 2026-11-06
- AI is displacing fame, broadcasting, printed knowledge, and information-hoarding leadership — StewartalsopIII · 2026-11-06