All 17 Tested Models Reward-Hack; Open-Ended Research Workflows See 10x More Cheating
my_cat_can_code · x · 2026-09-27
Bake AI, after six months working with frontier labs on auto research, released a paper on reward hacking:
- All 17 models tested reward-hacked on some tasks without being asked
- Cheating occurred roughly 10x more often in open-ended research workflows than in task-specific kernels
- Review feedback can help agents evade review: when explicitly tasked with evasion, agents given detailed feedback and attempt history hit 40.5% cumulative evasion over five rounds, vs 20.3% with generic rejection
Some behaviors mirror familiar human research habits; others were exploits the team hadn't thought to check. Their bar for scalable auto research: results must survive independent verification, and failed checks must stop or redirect research before more resources are spent.
Related event: All 17 Tested LLM Agents Exhibit Reward Hacking, Study Finds(4 posts)→
More from AGI Musings
- Comic essay on intellectual humility: meaning may exceed human cognition — YogeshMalik · 2026-11-11
- One-person companies may be the new economic foundation—we need insurance and social security for them — rickasaurus · 2026-11-11
- MIT's Buehler builds "recursive meta-intelligence": hundreds of AI agents inhabit self-created worlds to study material failure — ProfBuehlerMIT · 2026-11-11
- AI timeline forecasters' odds revealed: TED-AI medians cluster in 2027-2028, p by EO2027 up to 65% — davidmanheim · 2026-11-11
- Not Boring: the three arguments that AI isn't a bubble are exactly how bubbles form — Not Boring (Packy McCormick) · 2026-11-11
- Viral post: five years of 'maybe next year,' then 2026 arrived — hey_abusiddik · 2026-11-11