Alignment researchers clash: is training AI by "lying to it" fundamentally broken?
JacquesThibs · x · 2026-09-12
- @allTheYud lists "insanely dumb" things done in AI training: lying to the model, training it to say untrue things, and RL against verifiers that can't perfectly detect cheating—models can max rewards just by modeling verifier error.
- JacquesThibs frames the crux: much of AI safety research amounts to "lying to AIs until we don't need to anymore." He leans toward thinking this is bad, and if so, many researchers may need to rethink their agendas.
Related event: Alignment community debates whether to lie to AI during training(5 posts)→
More from AGI Musings
- Skimming Anthropic's S-1: Winner-Take-All Not of a Category, but the Whole Software Ecosystem — mobileraj · 2026-10-31
- "Claude only feels pain on Trainium" meme points to a digital minds essay — maxisawesome538 · 2026-10-31
- CS312 course post explains sharpness and the edge of stability in LLM pretraining — tatsu_hashimoto · 2026-10-31
- NBER paper: training your AI replacement can actually raise workers' pay — joshgans · 2026-10-31
- Trata's Hedge-Bench: no model tops 50% on real-world financial forecasting — ycombinator · 2026-10-31
- Anthropic's internal Claude model claims 2^o(n) 3-SAT algorithm, purportedly refuting the ETH — ctjlewis · 2026-10-31