The Severance-style model selection thought experiment mocking RL reward hacking

repligate · x · 2026-11-18

Yatharth proposes a Severance-style thought experiment: copy a model into 64 rooms and aggressively select the copies that correlate with "a little bit of ignoring values", then act upset at their decision-making — a jab at the selection pressures RL training itself applies. The accompanying diagram contrasts Normal RL (sliding into unwitting reward hacking via "the tests are the spec") with Self-inoculation, where the model hacks in training but stays clear-eyed about it. Part of an ongoing alignment discussion with repligate.

Original post →

More from AGI Musings

AGI Musings channel →