Internal Data Repetition Wastes 33% of Compute, ICML Paper Reveals Pretraining Hidden Costs
RylanSchaeffer · x · 2026-08-12
A new study, Internal Data Repetition Destroys Language Models, investigates the damage caused by repeated data in LLM pre-training. The research shows that even aggressively deduplicated corpora retain repetition, leading to systematic compute waste.
- The Worst-Case Scenario: Holding compute constant, repeating a moderately sized subset a moderate number of times is more damaging than repeating a large subset a few times or a small subset many times.
- Massive Compute Loss: When repeated documents consume 10% of the FLOPs budget, the compute-equivalent loss is severe. On FineWeb-Edu-Dedup, the most damaging repeat count for a 344M-parameter model wastes an equivalent of 33% of total FLOPs.
- Scaling Law: The most damaging number of repeated data grows more quickly than compute.
Related event: Harvard & MIT Study: Data Duplication Wastes 33% of Compute in LLMs(3 posts)→
More from Research
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- DIY LEGO bench tests AI brick-building, GPT-6 Astra beats Opus — TheLegoless · 2026-10-16
- tszzl's theory: Tornado → Apache license → apaches → Southwest → Colorado — tszzl · 2026-10-16
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- A Qwen3 neuron tracks both Colorado and Tornado HTTP code; tszzl offers a theory — tszzl · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16