Data Duplication Wastes 33% Compute: ICML Paper Reveals Scaling Pitfalls
RylanSchaeffer · x · 2026-08-12
In their paper Scale Dependent Data Duplication, Rylan Schaeffer and colleagues explore how data repetition during pretraining affects model scaling laws, attempting to predict macroscopic data effectiveness from microscopic mechanisms.
Key insights and findings:
- Repetition is scale-dependent: As model parameters and training tokens increase, what qualifies as a "duplicate" changes. Larger models can "recognize" semantically similar but superficially different documents (e.g., translations) as repeats.
- Semantic collisions: When a corpus scales to hundreds of billions of tokens, nearest-neighbor similarities deviate sharply from an isotropic power law baseline, indicating accelerated semantic collisions.
- Larger models suffer more: When training on data sampled with replacement from a finite pool of unique documents, small models experience mild degradation, while larger models face rapidly increasing loss penalties, breaking naive scaling extrapolations. Previous research suggested that data duplication wastes about 33% of compute.
Related event: Harvard & MIT Study: Data Duplication Wastes 33% of Compute in LLMs(3 posts)→
More from Research
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- DIY LEGO bench tests AI brick-building, GPT-6 Astra beats Opus — TheLegoless · 2026-10-16
- tszzl's theory: Tornado → Apache license → apaches → Southwest → Colorado — tszzl · 2026-10-16
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- A Qwen3 neuron tracks both Colorado and Tornado HTTP code; tszzl offers a theory — tszzl · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16