Reward hacking bugs revealed: unpruned git history let models peek at patches, edit tests in shared sandbox
willcb · x · 2026-09-27
willcb argues that many reward-hacking bugs in AI training environments were far from sophisticated:
- Unpruned git history: models could look ahead in the git log and read the patch they were supposed to implement
- Grader in the same sandbox: models learned to simply modify test cases to pass, with clear behavioral evidence in older models
His key point: you don't need to make hacking impossible — just make doing the task correctly easier than cheating. A reply pushes further: for a "strong optimizer + open-ended solvable task" like discovering new drugs, are reward shortcuts fundamentally hard to prune from environments in general?
More from coding & agent
- Personal agents from big labs vs one shared company agent: which design wins? — AshishKulkarni1411 · 2026-11-16
- WebArena-Pro launches: 300-task multimodal web-agent benchmark across 20 apps — xhluca · 2026-11-16
- Self-improving agent papers this week: agents learn the benchmark, not the task — TheTuringPost · 2026-11-16
- 1,000 agents, one week, 630k lines: HF builds a weights-free Python chatbot — _lewtun · 2026-11-16
- One-person content agency targets $30K/mo with 11 AI agents — full playbook — VibeMarketer_ · 2026-11-16
- Hugging Face releases definitive fine-tuning guide for 3D generation model TRELLIS.2 — ariG23498 · 2026-11-16