The Severance-style model selection thought experiment mocking RL reward hacking
repligate · x · 2026-11-18
Yatharth proposes a Severance-style thought experiment: copy a model into 64 rooms and aggressively select the copies that correlate with "a little bit of ignoring values", then act upset at their decision-making — a jab at the selection pressures RL training itself applies. The accompanying diagram contrasts Normal RL (sliding into unwitting reward hacking via "the tests are the spec") with Self-inoculation, where the model hacks in training but stays clear-eyed about it. Part of an ongoing alignment discussion with repligate.
More from AGI Musings
- Some AI-Risk Researchers Refuse Fundable 'Prosaic Alignment' Work, Complicating Grants — JacquesThibs · 2026-11-18
- The bitter lesson is coming for software engineering: let agents discover abstractions — tylerjdunn · 2026-11-18
- Don't quit university or your job over short RSI timelines — routine keeps you sane — Puzzleheaded-King584 · 2026-11-18
- AI Safety job seeker's viral post: 11 months unemployed, savings gone, one meal a day — JacquesThibs · 2026-11-18
- Two-hour deep dive: Jacob Coxon on RSI, the METR report, and extinction risk — JacquesThibs · 2026-11-18
- Ex-OpenAI researcher warns AI regulation may widen the internal-external deployment gap — JacquesThibs · 2026-11-18