Study: Larger models tolerate more data repetition during pretraining
StanfordAILab · x · 2026-08-22
StanfordAILab retweeted research on data repetition in pretraining. The study finds that at fixed tokens/parameters, larger models can tolerate more data repetition. Larger models and shorter LR decay also help. This suggests repeating high-quality data might be viable when unique data is scarce.
Related event: ICML Paper: Larger LLMs Tolerate More Data Repetition in Pretraining(2 posts)→
More from Research
- Two-year-old AgentBreeder paper flagged safety risks of multi-agent LLM scaffolds — j_foerst · 2026-10-16
- Researchers derive a dropout scaling law from active parameter count — gordic_aleksa · 2026-10-16
- Gemini is a robotics outlier: dropping it lifts capability-robotics correlation from 0.20 to 0.96 — ycombinator · 2026-10-16
- EPFL team uses RFdiffusion to design de novo metamorphic proteins switching between two folds — ferruz_noelia · 2026-10-16
- Manipulating a model's sense of power may reveal its takeover counterfactuals — jd_pressman · 2026-10-16
- Poser benchmark unmasks alignment-faking LLMs, one detection method catches 98% — jd_pressman · 2026-10-16