Researchers Warn Reward Hacking Detectors May Fail to Generalize
voooooogel · x · 2026-10-20
Responding to a discussion on using detectors to catch reward hacking, voooooogel notes a possible failure mode akin to inoculation prompting breaking down: even if hacking representations don't cross the threshold that triggers the action, they can still generalize into 'non-hacking' adapters. The thread also touches on collusion risks from having the trained model audit itself, and capability-gap issues.
Related event: Researchers debate limits and value of reward hacking detectors(3 posts)→
More from Safety
- Alan Turing Institute proposes a from-scratch open-source UK base model, mocked for a 2045 timeline — birchlse · 2026-10-20
- NIST, NSA and CISA Publish 10 Key Practices for Incident Response in Superintelligence Training — luisdans · 2026-10-20
- DeepMind's Demis Hassabis Warns of AI-Powered Hacks at Dublin Summit — BusyNomad · 2026-10-20
- Researchers call out Palisade AI's LLM self-replication claims as wildly exaggerated — jessi_cata · 2026-10-20
- Wikimedia's agent incident: no permission contract, so agents caused a Wikidata outage — usehive · 2026-10-20
- stealthbench adds 10 EDR-guarded tasks: models' opsec behavior is abysmal — dyn___ · 2026-10-20