kalomaze: Deduplicated DCLM still beats FineWeb as go-to web pretraining data
kalomaze · x · 2026-10-12
kalomaze argues that deduplicated DCLM remains the best "good enough" general webtext dataset for pretraining, and questions why FineWeb became the canonical choice for nanogpt speedruns — in his view it isn't even the best reheated-CommonCrawl option.
More from Research
- Scott Aaronson: the singularity has already begun for math — ChrSzegedy · 2026-10-12
- UN green chemistry brief highlights AI-designed enzymes beyond directed evolution — AnimaAnandkumar · 2026-10-12
- Nvidia paper: scoring the decisive passing edit predicts agent post-training payoff at 0.964 — rohanpaul_ai · 2026-10-12
- Maniformer paper cuts egocentric hand tracking error to 4.29mm for robot learning data — heyshrutimishra · 2026-10-12
- Breiman's 2001 'Two Cultures' paper resurfaces as AI makes black boxes the norm — akbirthko · 2026-10-12
- F. N. David's 1962 classic 'Games, Gods and Gambling' is now freely available as PDF — alexbilz · 2026-10-12