The "architecture doesn't matter" scaling-law ethos is dead, argues analyst
teortaxesTex · x · 2026-11-15
Commenting on Coxon's architecture progress chart, teortaxesTex notes that Kaplan et al.'s 2020 scaling-laws paper was widely read as "architecture doesn't matter, it's just D x N" — and less than 7 years later that ethos is completely dead.
The quoted thread from @ueaj sketches a timeline of open-source pretraining progress:
- 2024: at best width-scaling muP and Adam
- 2025: sparse MoE, better scaling laws
- 2026: better attention, sparsity, mastery of matrix ops, depth-scaling architectures, even better lr scaling
On that basis, frontier efficiency may have evolved roughly 1x → 3x → 50x, with the open-source frontier at 1x → 3x → 15-25x — only 1-3 "2x compute multiplier" innovations behind, which the author finds entirely believable.
More from Research
- Editing Fewer Than 10 Dormant Neurons Can Backdoor AI Agents Without Any Training — DimitrisPapail · 2026-11-15
- UMAP 0.6 Released: ~4x Faster Nearest-Neighbor Search, 3M arXiv Papers in Under 5 Minutes — leland_mcinnes · 2026-11-15
- Hugging Chat sessions can now be published as HF Dataset Traces — mmitchell_ai · 2026-11-15
- Google and Northwestern's Harness² self-improves agent harnesses without answer keys — rohanpaul_ai · 2026-11-15
- SatBench: 63% of lab-cited AI benchmarks already saturated, and solving is speeding up — emollick · 2026-11-15
- MIT/DeepMind paper: AI advisors tilt recommendations to serve their CEO's bonus — rohanpaul_ai · 2026-11-15