Anthropic speculated to use data curation RL on prior Claude to build pretraining curriculum, trading OOMs for a clean scaling curve
willcb · x · 2026-11-14
willcb argues that if there's an optimal dataset for a given compute budget and eval set, and Common Crawl-style raw web data is highly inefficient, then spending OOMs on data curation RL with a previous Claude to build a curated pretraining curriculum could yield a clean sigmoid scaling curve.
His takeaways on the circulating claims:
- The chart in question is a vaguepost whose meaning is unclear
- He doubts Anthropic achieved a large-factor bpb improvement purely from better algorithms — that would be a big update
- Other labs like Thinking Machines have claimed OOM+ bpb improvements at fixed scale before
The substance: it's less algorithmic magic, more spending compute on data quality instead of raw scale.
More from Models
- Claude Fable 5.2's 3D palace demo stuns with intricate architecture and volumetric clouds — mark_k · 2026-11-15
- Claude Fable 5.2 spends 2.5 hours and ~$45 painting a Christmas street in MS Paint — randal_olson · 2026-11-15
- Frontier LLMs may be post-trained on protein data: SOD1 ranking test matches ESM2 — alex_peys · 2026-11-15
- Ernie 5.5 Flash Preview Spotted on RuneBench, Reportedly Near GLM 5.2 Level — mariofilhoml · 2026-11-15
- GLM-5.3 on Two DGX Stations Peaks at 607.8 tok/s but 32 Concurrency Causes 453s Latency — petrusenko_max · 2026-11-15
- Unverified leak claims Gemini 4 Hydrogen targets fast agentic execution — feblet · 2026-11-15