Harvard & MIT Study: Data Reuse Leads to Diminishing Returns in LLM Training
burkov · x · 2026-08-13
LLM training is reaching a point where finding high-quality fresh text is harder than simply adding compute, while traditional 'scaling laws' largely assume fresh data can keep increasing with compute.
A recent preprint from Harvard and MIT examines a more realistic scenario: supplementing a fixed dataset with reused or paraphrased text. The authors introduce a metric to measure the value of these derived tokens compared to genuinely new ones.
Experiments show that the value of repeated data is not fixed: it yields diminishing returns as more is added. This provides a new perspective on how future training will be constrained by compute, fresh data, and the efficiency of data reuse.
Related event: Harvard & MIT Study: Data Duplication Wastes 33% of Compute in LLMs(3 posts)→
More from Research
- Quiz: Double the Residual Stream from 4096 to 8192 — How Does Param Count Change? — Sauers_ · 2026-10-16
- Researcher credits GPT for multiple accepted papers, incl. disproving Wegner's 1965 conjecture — ctjlewis · 2026-10-16
- Open-source shopper attention heatmap: YOLO11-pose + Gaze-LLE + Depth Anything V2 — claudioe · 2026-10-16
- AutoAgent0: VLM-powered agentic runtime lifts long-tail driving success by 28.3pp — yining_hong · 2026-10-16
- Shuffled labels hit AUC 0.59: selection bias, not signal, drives prosody benchmarks — maier_ak · 2026-10-16
- CODESTRUCT reframes codebases as AST action spaces to fix brittle LLM code agents — burkov · 2026-10-16