iamtrask: AI alignment is a data problem — every breakthrough is just better data
iamtrask · x · 2026-09-05
Researcher iamtrask argues that alignment is fundamentally a data problem: anything an AI knows, believes, or does can only come from data. The biggest alignment risks stem from unconstrained training data containing dangerous learning signal (especially web scrapes and RL reward-seeking), while the biggest breakthroughs — filtering, RLHF, reward shaping — are all just "better data" that erases or supplements the bad data. He adds that the biggest threat to verifiable alignment progress is secret natural data distributions.
More from AGI Musings
- Grady Booch revives Weinberg's 1971 quip: software so buggy one woodpecker could destroy civilization — Grady_Booch · 2026-10-23
- Why Netflix keeps losing to YouTube: UGC platforms professionalize easier than pro platforms go casual — ZeroStateReflex · 2026-10-23
- AI killed the coding moat: indie devs now compete on demand and distribution — sven_ai · 2026-10-23
- DynaVelo, a neural ODE model of single-cell dynamics, published in Nature Biotechnology — anshulkundaje · 2026-10-23
- Beff Jezos: Only frontier labs out-accelerating open source can secure systems — beffjezos · 2026-10-23
- 'I don't want to live in that world': a Redditor on why AGI utopia feels meaningless — peace_venerable · 2026-10-23