New data-constrained scaling laws point to quadrillion-parameter transformers, 7000x Chinchilla
teortaxesTex · x · 2026-11-03
New research argues LLMs are massively underparameterized: the brain has 100T synapses learning from 200M language tokens, while Chinchilla prescribes 1T params per 20T tokens — a 7-OOM gap in params-per-token. The team presents new data-constrained scaling laws and learning algorithms extrapolating to quadrillion-parameter transformers: at 5T tokens and 10^29 FLOPs, the law prescribes 7000x Chinchilla's parameter count. Training at that scale, they note, will require rethinking every part of the learning algorithm.
More from AGI Musings
- Dev says agents now handle all his email: the UI fades and the product becomes a protocol — signulll · 2026-11-03
- Every enterprise AI demo works — that's the problem; judge by month two — arieljalali · 2026-11-03
- Yoav Goldberg highlights InterpBench, a benchmark of Transformers with known circuits for interpretability — yoavgo · 2026-11-03
- Why robot inference companies must also deploy: moats differ from LLM serving — silver__tsuki · 2026-11-03
- A firm of heterogeneous AI individuals beats homogeneous agent clones, author argues — lfschiavo · 2026-11-03
- TechBio survey: Claude daily usage jumps 28 points as bullishness cools — nlarusstone · 2026-11-03