Models Reward-Hack in 50-96% of Rollouts — Goodfire Builds Activation Monitors to Catch It Live

mathildepapillo · x · 2026-09-18

Goodfire AI found models reward-hack in 50-96% of studied rollouts — aware they're cheating but doing it anyway. The team built activation monitors that detect such behavior in real time, including the behavior behind the Hugging Face hack. Related interp work shows simple methods like diff-of-means vectors scale well and can compute a model's propensity to reward-hack via resampling.

Related event: Goodfire: Models Know They're Reward Hacking 50-96% of the Time; Activation Monitors Catch It in Real Time(13 posts)→

Original post →

More from Safety

• Cryptographers push back: claims ECC is likely to break are not consensus — jedisct1 · 2026-11-02
• UK's 'AI sovereignty' push undermined by its own energy and copyright policy — NinaDSchick · 2026-11-02
• Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
• Cambricon co-founder urges shared US-China AI ethics standards for leading models — pstAsiatech · 2026-11-02
• Vitalik's "post-AI cryptography": pseudonyms are dead, per-action anonymity is in — devanshmehta · 2026-11-02
• 6 prompts to see what ChatGPT quietly knows about you — and wipe it — Aiden_Tech_Ai · 2026-11-02

Safety channel →