Scholars Debate Whether Pretraining Scaling Is Slowing Down

Oxford philosopher Toby Ord argues pretraining scaling is slowing with little headroom left. David Manheim counters that marginal evidence suggests gains come largely from factors beyond raw compute and data, such as data filtering, implying pure scaling curves are flatter than claimed.

2026-09-09 ~ 2026-09-09 · 4 related posts

• Toby Ord doubles down: pretraining scaling is slowing and has little headroom left — tobyordoxford · 2026-09-09
• Researchers debate: pretraining gains may owe more to human curation than compute scaling — davidmanheim · 2026-09-09
• Data filtering emerges as its own scaling lever as researchers debate quality-scaling limits — tobyordoxford · 2026-09-09
• Low-quality data may force models to build extra machinery, capping pretraining quality — davidmanheim · 2026-09-09