Stanford's Hinted Scaling Laws cut benchmark forecast error from 21 to 4.3 points
heghbalz · x · 2026-11-03
Stanford researchers (incl. Percy Liang and Tatsunori Hashimoto) released the arXiv paper "Hinted Scaling Laws," fixing a key failure of scaling-law extrapolation:
- Problem: extrapolating downstream accuracy from small models fails when they score near zero on hard benchmarks, leaving no trend to extrapolate
- Method: hinted scaling laws use privileged information (truncation, masking, logit mixing) to turn a benchmark into a continuum of easier versions indexed by hint level, so even small models score above zero and accuracy rises smoothly with hint strength
- Results: on the Delphi pretraining ladder, fitting only models up to 10^20 FLOPs forecasts a 10^23 FLOP model's GSM8K accuracy (25.9%) within 3 points; a standard law under-forecasts by 13 points. Across GSM8K/AIME/HLE, median forecast error drops from 21 to 4.3 points
- Bonus: hinting widens accuracy gaps, so one hinted rollout ranks models as well as five unhinted rollouts
Code and data are open-sourced (github.com/tatsu-lab/hinted-scaling-laws).
More from Research
- Christian Kroer's Book 'Games, Markets, and Online Learning' Now in Print, Free PDF Available — bremen79 · 2026-11-03
- Team Replicates Anthropic's "Teaching Claude Why" Alignment Method on Qwen and Open-Sources It — austinc3301 · 2026-11-03
- LoRA From Scratch: recovering a hidden weight change with two thin matrices — dima806_dima · 2026-11-03
- Contextual AI ships final contextual embedding models: 0.6B sharing 9B's embedding space — antoine_chaffin · 2026-11-03
- DLR-Lock proposes making open-weight LLM fine-tuning far costlier without breaking inference — burkov · 2026-11-03
- Free 30-min lesson: check if your model's 0.8 really means 80% — bgoncalves · 2026-11-03