ICML Paper: Larger LLMs Tolerate More Data Repetition in Pretraining

An ICML paper finds that larger language models tolerate more data repetition during pretraining at fixed tokens and parameters. The results suggest repeating scarce high-quality data can maintain tokens-per-parameter without wasting compute.

2026-08-22 ~ 2026-08-22 · 2 related posts

• Study: Larger models tolerate more data repetition during pretraining — StanfordAILab · 2026-08-22
• Paper Reveals Larger LLMs Tolerate More Data Repetition During Pretraining — heghbalz · 2026-08-22