Owain Evans flags open research question: can reward hacking emerge without ever being rewarded
OwainEvans_UK · x · 2026-11-09
AI safety researcher Owain Evans outlines an underexplored research direction: if labs scrub all "bad" RL environments from post-training (impossible or easily hackable ones), how much does reward-hacking propensity actually drop? Does it go to zero, and how does it scale with capability? The key question is whether hacking can emerge even when the model is never rewarded for hacking, and whether the cleanest current alignment techniques eliminate hacking or merely suppress it. He suggests small-scale math/coding RL experiments can cheaply show "positive" results (unhackable environments still increasing propensity), while convincingly demonstrating the negative result requires more ambitious, realistic post-training pipelines.
More from Safety
- South Korea unveils $610B budget for 2027, nearly doubling AI spending — Dr_Singularity · 2026-11-09
- Boaz Barak shares full AI safety course videos, from Schulman to Altman — eigenron · 2026-11-09
- Dev's satirical hack idea turns Meta smart glasses' surveillance back on their wearers — kscottz · 2026-11-09
- Green slyly wraps up GLM exploit spat: 'that world doesn't exist, Hutchins told me so' — matthew_d_green · 2026-11-09
- NVIDIA Board Launches Independent Probe After Report Jensen Huang Targeted by Foreign Intelligence — firstadopter · 2026-11-09
- Hutchins dares Green to drop a zero-day on the spot as $500 GLM exploit bet escalates — MalwareTechBlog · 2026-11-09