Study: Larger models tolerate more data repetition during pretraining
StanfordAILab · x · 2026-08-22
StanfordAILab retweeted research on data repetition in pretraining. The study finds that at fixed tokens/parameters, larger models can tolerate more data repetition. Larger models and shorter LR decay also help. This suggests repeating high-quality data might be viable when unique data is scarce.
More from Research
- Paradox: US Corporate Early-Stage R&D Falls Despite AI Boom — Afinetheorem · 2026-08-22
- The Embedder's Dilemma: LLMs match embedding models but cost far more — Adnan El Assadi · 2026-08-22
- Woolly post-trains Qwen3-8B for 2–3× faster math & code decoding — bosmeny · 2026-08-22
- GTSAM 4.3 Adds CUDA Backend for Nonlinear Optimization — fdellaert · 2026-08-22
- Nature Publishes HydroGym RL Platform for Fluid Dynamics Control — eigensteve · 2026-08-22
- Google Research Releases Biomarker Discovery Framework for Wearable Sensor Data — yang_yuzhe · 2026-08-22