EMNLP Paper: Repetition Mismatch Explains Why Data Mixture Experiments Don't Scale
MarekRei · x · 2026-10-25
A paper by Imperial College London and Cohere, presented at EMNLP 2026 in Budapest, identifies "repetition mismatch" as a primary cause of small-scale data mixture extrapolation failure in pretraining.
- When high-quality data is limited, larger token budgets change repetition rates, shifting the optimal mixture — so small-scale experiments mispredict full-scale results.
- The fix (repeat-aware): subsample high-quality data at each proxy horizon to match target repetition rates, compared against scaling-law baselines using FineWeb, WikiText and PubMed with models up to 3.48B tokens target budget.
- Results: repeat-aware with one horizon is closer to optimal at every model size (e.g. 0.10 vs 0.85 error at 1.17B on WikiText); multiple horizons beat scaling laws at 1.17B, at rising compute cost (6%/19%/44%/94% extra tokens per added horizon).
- Takeaway: repetition should be an explicit variable in mixture optimization; existing methods can adopt resampling directly.
More from Research
- Training models locked to a single GPU type could stop exfiltrated agents from self-migrating — andersonbcdefg · 2026-10-25
- Economists debate whether AI-era economics will produce 10x new economists or just more empiricism — soumitrashukla9 · 2026-10-25
- Imperial NLP Brings 6 Main Papers and Agent-LM Workshop to EMNLP 2026 — MarekRei · 2026-10-25
- Google's Jon Barron shares his full AI paper-writing workflow with Gemini — lmoroney · 2026-10-25
- Only 30% of suspected rare-disease newborns get diagnosed; Gamow Labs launches 122-case RareBench for agentic AI — Sauers_ · 2026-10-25
- MMGR benchmark: top video models solve only ~3% of mazes despite stunning visuals — zhaofeng_wu · 2026-10-25