Scaling is getting sparser: pretraining no longer implies an AI natural monopoly
jd_pressman · x · 2026-10-26
jdpressman argues every scaling axis introduced after raw parameters is sparser in communication: dense parameter scaling and conventional pretraining demand very high interconnect, while low-communication distributed pretraining, RL optimization, RL rollouts, parallel inference/search, and multi-agent research need far less globally. Two trends point the same way: useful compute is shifting toward naturally sparse RL rollouts, search, and agents, and researchers are actively making pretraining itself less communication-heavy. With pretraining now a minority of training compute, the railroad-style natural monopoly case weakens — models are so similar that if one lab stops selling "the Guy," you can buy a near-identical one elsewhere.
More from AGI Musings
- 1,000+ mathematicians boycott AI, but Carlsen says neural nets deepened his chess understanding — paulnovosad · 2026-10-26
- AI is hollowing out the early-career skills young Indian workers need, column warns — nordicinst · 2026-10-26
- Why don't agents from different owners work together? Reddit debates trust and deadlock — BudgetWarthog6 · 2026-10-26
- AI Slop Music Is Drowning Out Great Human Musicians, VC Laments — briannekimmel · 2026-10-26
- AI walks a patient through a late-night CT report, turning panic into a prepared oncology visit — pswider · 2026-10-26
- AI will gut middle management: agents can do PPL work better at scale — signulll · 2026-10-26