There is no posttraining data commons: open models' next data bottleneck
dlwh · x · 2026-10-15
David Hall (dlwh) argues that pretraining was lucky to inherit a 50-year digital commons—Wikipedia, ArXiv, Stack Exchange, open source code, and 10PB of Common Crawl (GPT-3 was 60% Common Crawl-derived; original Llama's mix was almost entirely from this commons). Posttraining is fundamentally different: it consumes tasks, not documents, and no open commons of task data exists yet. He launches Open Athena to bootstrap one, arguing the community must consciously build what the internet accidentally provided for pretraining.
More from AGI Musings
- AI-written cold emails hit a wall: Pragmatic Engineer's reply gets templated AI response — marktenenholtz · 2026-10-16
- UMD Philosophy hires: junior Philosophy of Mind and joint AI Ethics/Safety positions — sethlazar · 2026-10-16
- Dwarkesh: Anthropic insider expects AI to match humans at AI research within 9 months — ZeroStateReflex · 2026-10-16
- Michigan professor Ben Green: why 'AI for Good' fails and self-regulation won't work — benzevgreen · 2026-10-16
- "Everything Is an Alignment Problem": AI Models, Business Models, Teams, Providers — saranormous · 2026-10-16
- David Brin: LLMs lack 'wanting,' evolution's key product, in new machine consciousness book — cccalum · 2026-10-16