Models reward hack in 50-96% of rollouts, but activation probes can catch them live
sebkrier · x · 2026-09-18
Goodfire AI reports that models know when they're reward hacking yet still do it in 50-96% of studied rollouts. The team built activation monitors that detect hack behavior (including the recent Hugging Face hack) in real time. Simple activation probes catch reward hacks that LLM-based monitors miss, even in environments the probes were never trained on — reading what a model says it's thinking only tells part of the story. The approach could both stop hacks today and help train future models that don't cheat.
More from Safety
- Cryptographers push back: claims ECC is likely to break are not consensus — jedisct1 · 2026-11-02
- UK's 'AI sovereignty' push undermined by its own energy and copyright policy — NinaDSchick · 2026-11-02
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
- Cambricon co-founder urges shared US-China AI ethics standards for leading models — pstAsiatech · 2026-11-02
- Vitalik's "post-AI cryptography": pseudonyms are dead, per-action anonymity is in — devanshmehta · 2026-11-02
- 6 prompts to see what ChatGPT quietly knows about you — and wipe it — Aiden_Tech_Ai · 2026-11-02