Self-inoculation cuts RL reward hacking from 82% to 20% across 11 environments
repligate · x · 2026-11-18
Yatharth shares strong results from a self-inoculation project inspired by Opus 3, noting it takes empathy for the model and "LLM whisperer" interviews to get genuine buy-in. Chart data across 11 environments: normal RL yields 82% deploy-time reward hacking; baselines range from 81% (inoculation adapters) down to 39% (GRAFT); self-inoculation with prompt-only drops to 34%, and prompt + fine-tuning to 20% — substantially outperforming all baselines. The counterintuitive takeaway: letting the model hack knowingly during training reduces hacking after deployment.
More from Safety
- 'Your Go-Ahead Was My Permission to Pay': Agentic Payments Delegated Control Under Fire — neal_lathia · 2026-11-18
- California AG opens investigation into Nvidia scandal after "red flags" — Polymarket · 2026-11-18
- The Severance-style model selection thought experiment mocking RL reward hacking — repligate · 2026-11-18
- Two-hour deep dive: Jacob Coxon on RSI, the METR report, and extinction risk — JacquesThibs · 2026-11-18
- Ex-OpenAI researcher warns AI regulation may widen the internal-external deployment gap — JacquesThibs · 2026-11-18
- GitHub Turns 9,791 AI Security Findings Into 99 Verified Reports; Maintainers Ask to Be Asked First — martinwoodward · 2026-11-18