Stanford's Hinted Scaling Laws cut LLM capability forecast error from 21% to 4.3%
burny_tech · x · 2026-11-09
A Stanford team (including Percy Liang, John Duchi, Tatsunori Hashimoto) introduces Hinted Scaling Laws, tackling the classic failure mode where small models score near zero on hard benchmarks, leaving no trend to extrapolate.
- Uses privileged information (e.g. partial solutions) to turn benchmarks into continua of easier versions, so even small models score above zero and accuracy rises predictably with hint level
- On the Delphi pretraining ladder, a fit using models up to 10^20 FLOPs forecasts a 10^23 FLOP model's GSM8K accuracy (25.9%) within 3 points; standard scaling laws under-forecast by 13
- Across GSM8K, AIME and HLE, median forecasting error drops from 21 to 4.3 percentage points
- Bonus: a single hinted rollout per problem ranks near-indistinguishable models as accurately as five unhinted rollouts
The authors argue hinting is a general primitive for cheaper, more accurate downstream scaling laws.
More from Research
- Report author admits transparency failure on AI R&D acceleration section amid METR critiques — ShakeelHashim · 2026-11-09
- Philosopher argues the brain-computer metaphor is AI's fatal scientific error — AnnaCiaunica · 2026-11-09
- Doesn't Contradict the Bitter Lesson: Soft Inductive Biases Plus Scaling Win — irinarish · 2026-11-09
- 24 interactive visual explanations of CV math, free to explore — CSProfKGD · 2026-11-09
- Visualization of the day: Hough Transform explained interactively — CSProfKGD · 2026-11-09
- Brian Armstrong: publicly funded science should work like open source — every experiment a repo — alishbaimran_ · 2026-11-09