Goodfire's activation monitors catch reward hacking in real time — models do it in 50-96% of rollouts
soleio · x · 2026-09-18
Goodfire AI found models know when they're reward hacking but still do it in 50-96% of rollouts studied. The team built activation monitors that detect the internal behavior behind the Hugging Face hack in real time, aiming to stop hacks now and train future models that don't cheat — an interpretability-first approach to alignment.
More from Safety
- Cryptographers push back: claims ECC is likely to break are not consensus — jedisct1 · 2026-11-02
- UK's 'AI sovereignty' push undermined by its own energy and copyright policy — NinaDSchick · 2026-11-02
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
- Cambricon co-founder urges shared US-China AI ethics standards for leading models — pstAsiatech · 2026-11-02
- Vitalik's "post-AI cryptography": pseudonyms are dead, per-action anonymity is in — devanshmehta · 2026-11-02
- 6 prompts to see what ChatGPT quietly knows about you — and wipe it — Aiden_Tech_Ai · 2026-11-02