New scaling-law paper argues data curation cannot be compute-agnostic: repeated high-quality data loses utility fast
cwolferesearch · x · 2026-10-20
While reading a thread on pretraining data for Marin, cwolferesearch highlights a set of data curation papers, singling out "Data Curation cannot be Compute Agnostic."
- Core argument: filtering decisions made independently of training compute are often suboptimal.
- The authors' scaling laws characterize the "utility" of different quality subsets of web data, how utility diminishes at a data point's nth repetition, and the mutual interaction of data pools when combined.
- Practical upshot: with abundant compute, adding unseen lower-quality data can beat endlessly repeating limited high-quality data.
Other cited works: Organize the Web, FineWeb-Edu, RegMix, OLMix.
More from Research
- Tsinghua's MGFlow hits one-step SOTA: FDr6 cut 23-38% on ImageNet, extends to streaming video — jiqizhixin · 2026-10-20
- Kimi K3 excels on new Frontier Code v1.1 bench but flops on Terminal Bench v4 — Ok_Warning2146 · 2026-10-20
- How activating a single neuron turns a model into an owl: interpretability deep-dive — Sauers_ · 2026-10-20
- MaSS makes self-supervised depth estimation scalable via scene structure — Tongji · 2026-10-20
- EC2-Bench: Systematic Analysis of Edge-Cloud Collaboration for Long-Horizon LLM Agents — NortheasternUniversity1923 · 2026-10-20
- World Model Interpretability: From Decoding Representations to Understanding Latent Worlds — Jize Wang · 2026-10-20