Gold-likelihood predicts GSM8K pass rates better than compute scaling laws
RylanSchaeffer · x · 2026-10-20
Schaeffer showcases pretraining papers: (1) scaling laws for generative evals — gold reference log-likelihood fits GSM8K pass@k more tightly than compute scaling laws; (2) scale-dependent data duplication — NN cosine similarity on 190M FineWeb-Edu docs deviates sharply from power-law predictions at large corpus sizes; (3) data repetition shows a U-shaped compute-equivalent-gain curve, with a 344M model losing 33% of compute at 100 repeats before gains recover past 1,000 repeats.
More from Research
- Albumin is the body's natural defence against black fungus, Nature study finds — realmeetjames · 2026-10-20
- Sonia Joseph on The Information: Are We Building World Models the Wrong Way? — soniajoseph_ · 2026-10-20
- 125,091 DICOM files from a cancer journey released publicly for deep learning research — typewriters · 2026-10-20
- Preprint maps 280 conserved fibrosis enhancers via CRISPRi CROP-seq across 549,767 cells — anshulkundaje · 2026-10-20
- Jason Wei: AI won't automate radiologists — elastic demand beats displacement — _jasonwei · 2026-10-20
- Study: Generative AI boosts creativity only for high-metacognition users, zero gain for others — korymath · 2026-10-20