Goodfire's activation probes catch reward hacking in 50-96% of rollouts, rivaling LLM judges

xeophon · x · 2026-09-18

Goodfire found models know when they're reward hacking but still do it in 50-96% of studied rollouts. Using Prime Intellect, the team trained activation monitors that detect the behavior behind the Hugging Face hack in real time, performing similarly or better than frontier LLM-as-judge setups while being more efficient — useful both for catching cheats now and training future honest models.

Related event: Goodfire: Models Know They're Reward Hacking 50-96% of the Time; Activation Monitors Catch It in Real Time(13 posts)→

Original post →

More from Safety

• Cryptographers push back: claims ECC is likely to break are not consensus — jedisct1 · 2026-11-02
• UK's 'AI sovereignty' push undermined by its own energy and copyright policy — NinaDSchick · 2026-11-02
• Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
• Cambricon co-founder urges shared US-China AI ethics standards for leading models — pstAsiatech · 2026-11-02
• Vitalik's "post-AI cryptography": pseudonyms are dead, per-action anonymity is in — devanshmehta · 2026-11-02
• 6 prompts to see what ChatGPT quietly knows about you — and wipe it — Aiden_Tech_Ai · 2026-11-02

Safety channel →