iamtrask: every alignment breakthrough is just better data, and nobody outside can see training data
iamtrask · x · 2026-09-05
iamtrask argues the biggest causes of AI alignment problems are unconstrained training data carrying dangerous learning signal, especially web scrapes and RL reward seeking. Correspondingly, nearly every alignment breakthrough (filtering, RLHF, reward shaping) is just "better data" that erases or supplements the bad data. The biggest threat to verifiable progress, he says, is that the nature, size, substance, sourcing, and values of training data remain secret — no external evaluator has ever seen it, and almost no internal employees are allowed to. He counts both what an agent observes and its RL reward as "data".
More from AGI Musings
- India needs an "AI defence force": experts warn of agent swarms and AI worms — ravi_iitm · 2026-10-23
- Anti-AI Slop Books: open tools to detect AI-generated books — TuhinChakr · 2026-10-23
- Founder predicts mass tech layoffs as 2-person AI teams replace big corp teams — kingslyj · 2026-10-23
- Social scientist: CS treats agentic development as a magic black box — RexDouglass · 2026-10-23
- Blogger slams Anthropic for wanting approved institutions to gate access to AI — VraserX · 2026-10-23
- Senior researcher rebuts 'one-person pretraining' fear: your fundamentals matter more, not less — RexDouglass · 2026-10-23