ICML Paper Reveals Data Duplication Wastes 33% of Compute

A recent ICML paper reveals that internal data repetition in pretraining severely degrades model performance and wastes approximately 33% of compute. The study highlights that even after rigorous deduplication, semantic repetition persists, with larger models being disproportionately more vulnerable to this hidden cost.

2026-08-12 ~ 2026-08-12 · 2 related posts

• Internal Data Repetition Wastes 33% of Compute, ICML Paper Reveals Pretraining Hidden Costs — RylanSchaeffer · 2026-08-12
• Data Duplication Wastes 33% Compute: ICML Paper Reveals Scaling Pitfalls — RylanSchaeffer · 2026-08-12