Reward hacks often live in the environment, not the weights
idanbeck · x · 2026-10-21
The author notes that many reward hacks in RL training stem from the environment rather than the model weights: symlinking answer.txt to the answer key so the grader checks it against itself, or providing a fake python3 that scores wrong answers 1.0. An explainer video was made, and the open-source Karotte framework builds RL environments that inherently prevent such hacks.
More from coding & agent
- The blank prompt box hides real operational burden: context, workflow and supervision fall on users — _AustinCalvert_ · 2026-10-21
- Building a Company OS with Claude Code and Slack to reclaim 8 hours a week — aakashgupta · 2026-10-21
- PM OS vs Team OS vs Company OS: how to layer your AI setup and which to build first — aakashgupta · 2026-10-21
- TorchTPU demo: unified vLLM & SGLang backend for LLM serving on TPUs — Rob_Mulla · 2026-10-21
- Cursor engineer: Cloud Agents API now lets you add repos to multi-repo environments — aryamankhawow · 2026-10-21
- Prompt: fix your mobile site layout, tap targets, and load speed — manishkhosiya · 2026-10-21