'Most algorithmic progress is data progress': why data quality beats clever algorithms
teortaxesTex · x · 2026-11-21
teortaxesTex shares an essay titled "Most Algorithmic Progress is Data Progress" and uses it to criticize China's AI ecosystem for being weak on data, noting "DeepSeek has to do this internally."
The essay argues that decomposing AI progress into compute + 'algorithmic progress' is misleading — the remainder is mostly data progress:
- Truly impactful algorithmic wins are rare (RLVR, mid-training, both dataset-dependent); qk-norm, fine-grained experts, and muon are dwarfed by better data.
- A 'tiger-parenting' analogy: Qwen3 trains on vast synthetic math/code data targeting benchmarks, while Llama2 saw mostly uncurated web pages — the latter wins at irrelevant skills like knowing 2008 forum posts.
- Evidence: ScaleAI and SurgeAI each rake in $1B revenue, mostly from frontier-lab labeling, possibly growing 3-4x per year like compute spend.
- Benchmark gains should be read as billions paid to PhDs/MDs to write questions and example answers, not just scale or clever research.
More from AGI Musings
- Singularity While North Sentinel Islanders Still Hunt With Bows — willdepue · 2026-11-21
- Can AI Do Embodied Jobs Like Medicine Without Having a Body? — midwoodgirl10 · 2026-11-21
- Claimed Riemann hypothesis proof co-authored with Claude stuns math community — zetalyrae · 2026-11-21
- zetalyrae on the Riemann Proof: Viral Essay Slams AI Safety's Claude-Centrism — zetalyrae · 2026-11-21
- A Skeptical Take on AI Takeover Scenarios: Seizing Hard Power Is Harder Than Suggested — sayashk · 2026-11-21
- jd_pressman: Technical Alignment Is Probably Easier Than the Riemann Hypothesis — jd_pressman · 2026-11-21