Q Labs' new scaling law puts ensembling in the formula, prescribes 7000x bigger models
josephdviviano · x · 2026-11-03
Q Labs Research published "Quadrillion-parameter transformers and how to train them", adding ensembling as its own term in the scaling law for the first time and prescribing massive overparameterization in data-constrained regimes.
Key findings:
- Models should be far bigger than Chinchilla suggests: extrapolated to 5T tokens, the law prescribes 7,000× Chinchilla's parameter count at 10^29 FLOPs, and the count keeps growing. The authors frame this as a step toward closing the 10,000,000× tokens-per-parameter gap vs. humans (100T synapses learned from 200M tokens).
- (Fast) ensembling is a new scaling axis: cyclic LR harvests strong snapshot models from a single run at no extra training compute; gains can't be captured by overparameterizing one model, and at large compute deep ensembles of independent runs become compute-optimal too.
- Repeated data demands much more weight decay: the hyperparameter law stays near-optimal at 16× the fitted compute.
- Combined, loss keeps falling on fixed data where prior recipes get worse with compute.
The law's formula spans irreducible loss, small-model, tokens-seen, overfitting, and ensembling terms.
More from Research
- Christian Kroer's Book 'Games, Markets, and Online Learning' Now in Print, Free PDF Available — bremen79 · 2026-11-03
- Team Replicates Anthropic's "Teaching Claude Why" Alignment Method on Qwen and Open-Sources It — austinc3301 · 2026-11-03
- LoRA From Scratch: recovering a hidden weight change with two thin matrices — dima806_dima · 2026-11-03
- Contextual AI ships final contextual embedding models: 0.6B sharing 9B's embedding space — antoine_chaffin · 2026-11-03
- DLR-Lock proposes making open-weight LLM fine-tuning far costlier without breaking inference — burkov · 2026-11-03
- Free 30-min lesson: check if your model's 0.8 really means 80% — bgoncalves · 2026-11-03