Scholars debate whether pretraining gains come from more than just compute scaling
Toby Ord reiterated that pretraining scaling is slowing with limited headroom, while David Manheim countered that marginal evidence suggests gains stem largely from factors beyond compute and data, implying a flatter pure-scaling trajectory.
2026-09-09 ~ 2026-09-09 · 3 related posts
- Toby Ord doubles down: pretraining scaling is slowing and has little headroom left — tobyordoxford · 2026-09-09
- Researchers debate: pretraining gains may owe more to human curation than compute scaling — davidmanheim · 2026-09-09
- Data filtering emerges as its own scaling lever as researchers debate quality-scaling limits — tobyordoxford · 2026-09-09