Pretraining insider notes: data repetition and ~1000 TPP overtraining are standard practice
SonglinYang4 · x · 2026-11-11
A candid thread of "unpopular-but-true" pretraining research notes, endorsed by researcher Songlin Yang:
- Data repetition is a very common practice, not an exception.
- Over-training (1000 tokens per parameter) is likewise standard.
- In the long run, data-efficient architectures and algorithms will matter more than hardware-friendly ones.
- FineWeb-Edu is a poor dataset for testing architectures — results from it can be misleading.
Useful context for anyone interpreting pretraining papers or running architecture comparisons.
More from Infra
- Apple reportedly building Harbor app to turn Macs into private AI inference servers with Thunderbolt 5 clustering — BenBajarin · 2026-11-11
- Agent Arena: Gemini 4 Carbon Gets More Cost-Competitive as Tasks Get Harder — rseroter · 2026-11-11
- AMD Zen 6 deep dive: cheaper branch mispredicts and a return to Zen 3/4 integer scheduling — jedisct1 · 2026-11-11
- Together AI SVP Dan Fu on kernels, serving 3T-param models day zero, and self-improving inference — togethercompute · 2026-11-11
- ExecuTorch adds WebGPU backend: run PyTorch models in-browser, up to 10x faster than CPU — PyTorch · 2026-11-11
- Hedra runs training experiments with AI agents steered by human researchers — testingcatalog · 2026-11-11