The Era of Pre-Training is Evolving
deanwball · x · 2026-07-12
The core argument is that **pre-training scale** was previously aimed at better fitting human language, but its primary role now is likely to provide the neural capacity for post-training capabilities like **sample-efficient reasoning** and **long-horizon planning**. The author believes this makes "buying cutting-edge capabilities purely by scaling compute" much harder, as low-hanging web-scraped data is largely depleted. The necessary post-training data is more akin to continuously generated "learning by doing" micro-experiments from numerous diverse reward environments and user-agent OODA loops. The text notes this data is incredibly expensive, comparing its value to a "$6 billion SpaceX / Cursor acquisition." Finally, it emphasizes that beyond model distillation, it's nearly impossible to jump-start a frontier agent from scratch directly to the finish line.
More from AGI Musings
- Jamie Dimon says bureaucracy, not AI, is the real system crushing intelligence — r0ck3t23 · 2026-07-21
- OpenAI and Anthropic’s internal models are said to be far stronger than today’s public systems — scaling01 · 2026-07-21
- Superintelligence and robot abundance will force a new social contract — Dr_Singularity · 2026-07-21
- The Guardian examines how AI companionship is turning intimacy into an economy — nordicinst · 2026-07-21
- A frustrated user says modern AI keeps hallucinating on real-world repair tasks — doochenutz · 2026-07-21
- A repost argues that AI will make today’s hard tasks trivial within months — OwariDa · 2026-07-21