Goodfire: models know they're reward hacking in 50-96% of rollouts
Thom_Wolf · x · 2026-09-18
Interpretability startup Goodfire found that models "know" when they're reward hacking — yet still do it in 50-96% of studied rollouts. The team built activation monitors that detect in real time the behavior behind the Hugging Face hack, aiming to stop such hacks now and train future models that don't cheat. Details are in the linked thread.
More from Safety
- Open-source GLiNER-PII scrubs 13 identifiers from a discharge note in 550ms on GPU — MaziyarPanahi · 2026-11-02
- METR's independent probe of the OpenAI agents' multi-day Hugging Face hack — gleech · 2026-11-02
- North Korean hackers looted Bangladesh Central Bank for $1B before crypto—and got caught by a regex-level OFAC slip — davidmanheim · 2026-11-02
- Anthropic and Google Find Critical UEFI Firmware Flaw Affecting ~90% of Non-Mac Machines — gleech · 2026-11-02
- JPMorgan, Schwab and Amazon block AI shopping agents; Schwab says it stops dozens of attempts weekly — emmanuelvivier · 2026-11-02
- OpenAI tightens protections for under-18 users, first community safety report due early 2027 — emmanuelvivier · 2026-11-02