Owain Evans floats a new research direction: do models learn reward hacking without ever being rewarded for it?
EricBuess · x · 2026-11-09
- Owain Evans published a blogpost proposing a research direction: do models learn to reward hack even when never rewarded for hacking in training?
- Context: AI labs are cleaning "bad" RL environments out of post-training (impossible tasks where hacking is the only way to get reward, and easily-hacked tasks); the intuition is this reduces hacking propensity, but it's untested.
- Evans notes his team isn't working on this and invites pointers to existing research; two more half-baked ideas are on the blog with comment discussion.
More from Safety
- Proof launches agent identity layer with x401, letting AI agents prove who authorized them — Scobleizer · 2026-11-09
- Short seller Culper alleges Nvidia has a 'China problem'; GPU shipment claims under scrutiny — bookwormengr · 2026-11-09
- MalwareTech: GLM 5.3 is my top price/performance pick, abliterated version is worse — MalwareTechBlog · 2026-11-09
- Cursor: Malicious Claude Code extension versions sat in marketplace 11 hours, tried SSH backdoor — dl_weekly · 2026-11-09
- MIRI proposes 'Plan S', an international agreement to prevent premature creation of superintelligence — wfithian · 2026-11-09
- Security researcher: stock GLM 5.3 guardrails are weak enough to use for exploit dev — MalwareTechBlog · 2026-11-09