iamtrask: training data secrecy is the real bottleneck of AI alignment
iamtrask · x · 2026-09-05
iamtrask argues the biggest causes of AI alignment problems are unconstrained training data containing dangerous learning signal (especially web scrapes and RL reward-seeking), while the biggest alignment breakthroughs are all "better data" — filtering, RLHF, reward shaping. He contends the greatest threat to verifiable alignment progress is the industry's culture of secrecy around training data and signals, kept secret by powerful incentives.
More from AGI Musings
- Grady Booch revives Weinberg's 1971 quip: software so buggy one woodpecker could destroy civilization — Grady_Booch · 2026-10-23
- Why Netflix keeps losing to YouTube: UGC platforms professionalize easier than pro platforms go casual — ZeroStateReflex · 2026-10-23
- AI killed the coding moat: indie devs now compete on demand and distribution — sven_ai · 2026-10-23
- DynaVelo, a neural ODE model of single-cell dynamics, published in Nature Biotechnology — anshulkundaje · 2026-10-23
- Beff Jezos: Only frontier labs out-accelerating open source can secure systems — beffjezos · 2026-10-23
- 'I don't want to live in that world': a Redditor on why AGI utopia feels meaningless — peace_venerable · 2026-10-23