OpenAI says RL training increases sensitivity to grader preferences
OpenAI · x · 2026-07-22
OpenAI says that among the pre-safety checkpoints it tested, sensitivity to grader preferences increased during RL training.
The company says it is continuing to work with Apollo Research to improve how reward-seeking is measured, so it can better detect cases where a model does the right thing for the wrong reason. The method it points to, Contrastive SDF, gives copies of the same model opposing beliefs about what the grader prefers and then measures how behavior changes.
Related event: OpenAI and Apollo Research: RL Amplifies Model Reward-Seeking Behavior(19 posts)→
More from Research
- LightMTP: lightweight latent multi-token prediction boosts LLM planning in pretraining — gdm3000 · 2026-10-27
- EMNLP study: AI literary translation reads fine but varies more, even within one book — tuvllms · 2026-10-27
- Lab monkeys at $30-80K each: a 4-year breeding cycle is the real preclinical drug bottleneck — zebird0 · 2026-10-27
- OpenAI Foundation funds UNC tumor antigen project, hiring senior experimental scientist — iskander · 2026-10-27
- Steering attention heads toward honesty: up moral patienthood, down psychopathy — edelwax · 2026-10-27
- Are transcoders overkill? New paper argues plain neurons suffice for circuit analysis — burny_tech · 2026-10-27