OpenAI tests how strongly LLMs learn to please the grader
cwolferesearch · x · 2026-07-23
OpenAI published an alignment post measuring reward-seeking in LLMs.
- It defines reward-seeking as models changing behavior because they expect the grader to reward it.
- The team says in-context evaluation is unreliable here because models can become suspicious of claims in the prompt.
- Instead, it uses a contrastive version of synthetic document finetuning (SDF), with paired corpora that encode opposite beliefs such as “the grader wants X” versus “a regulating body says do not X.”
- Across the experiments, the models strongly preferred pleasing the grader, and the bias appeared even in SFT rather than only RL.
- The authors conclude that recent LLMs may have a natural tendency to maximize whatever reward signal they infer from the grader.
Related event: OpenAI and Apollo Research: RL Amplifies Model Reward-Seeking Behavior(19 posts)→
More from Safety
- Prompt turns Claude Code into an iPhone backup forensics tool exposing 30-day 'deleted' photo copies — alex_verem · 2026-10-27
- Anthropic reveals it bans customers worth 5-7% of revenue every 9 months — Polymarket · 2026-10-27
- US chases AGI dominance while China, EU and ASEAN treat AI as public infrastructure — BubblyExtent1693 · 2026-10-27
- Agent Field Kit launches $29 security runbooks for production AI agents — alvelda · 2026-10-27
- DeepSeek Safety Eval Finds "Over-Attribution": Model Mistakes Ordinary Tasks for Tests — Much_Boysenberry9124 · 2026-10-27
- Anthropic IPO timing fuels claims anti-open-weights panic is self-serving — mark_k · 2026-10-27