Scaling law ambiguity was settled in NeurIPS 2022, Google researchers' paper resurfaces
anshulkundaje · x · 2026-11-06
giffmana points out that the recent scaling law controversy was already addressed in the NeurIPS 2022 paper 'Revisiting Neural Scaling Laws in Language and Vision' (arXiv:2209.06640) by Google researchers. The paper argues for evaluating scaling laws via extrapolation loss rather than best-fitting (interpolating) parameters: many scaling exponents fit learning curves almost equally well, but the best-fitting parameters don't necessarily yield accurate extrapolation, so a rigorous recipe is needed to estimate scaling parameters. It also released a benchmark of 90 evaluation tasks. Lilian Weng's scaling laws blog widget and Andreas's blogpost both reflect this ambiguity.
More from Research
- Zhipu chief scientist reveals GLM pretraining data roadmap: GLM-5 at 28.5T tokens, 100T next — rickasaurus · 2026-11-06
- Laude Institute grants $10M to UChicago team building AI weather forecasts for farmers — ChenhaoTan · 2026-11-06
- Quick test: Gemini 3.1 Pro knows the answer in its weights, Flash doesn't — giffmana · 2026-11-06
- JupyterHealth team wins $10M Laude Institute Moonshot Prize for open-source health data infra — irenetrampoline · 2026-11-06
- SmoothScore: SOTA LLMs still 'smooth' nuanced research findings into overclaims — gregd_nlp · 2026-11-06
- Schmidhuber: Yale's fast weight programming course reprises his 1990s linear Transformer work — SchmidhuberAI · 2026-11-06