iamtrask: AI alignment breakthroughs are all just 'better data'—and that's the real issue
iamtrask · x · 2026-09-05
In a long thread, researcher iamtrask argues that the biggest causes of AI alignment problems are unconstrained training data with dangerous learning signal (web scrapes, RL reward seeking), and that every major alignment breakthrough—filtering, RLHF, reward shaping—is fundamentally 'better data.' He warns that unexamined 'secret natural data' is the biggest threat to verifiable alignment progress, and claims a massive information campaign distracts from this core issue.
More from AGI Musings
- Grady Booch revives Weinberg's 1971 quip: software so buggy one woodpecker could destroy civilization — Grady_Booch · 2026-10-23
- Why Netflix keeps losing to YouTube: UGC platforms professionalize easier than pro platforms go casual — ZeroStateReflex · 2026-10-23
- AI killed the coding moat: indie devs now compete on demand and distribution — sven_ai · 2026-10-23
- DynaVelo, a neural ODE model of single-cell dynamics, published in Nature Biotechnology — anshulkundaje · 2026-10-23
- Beff Jezos: Only frontier labs out-accelerating open source can secure systems — beffjezos · 2026-10-23
- 'I don't want to live in that world': a Redditor on why AGI utopia feels meaningless — peace_venerable · 2026-10-23