Research: Reverse-Engineering Black-Box Rewards to Understand LLM Learning
sanmikoyejo · x · 2026-07-05
@edchene released a paper focusing on "what behaviors black-box reward functions actually align LLMs to." The authors note that aligning with uninterpretable black-box rewards obscures the model's actual learned objectives, potentially introducing hidden misalignments. They propose a detection method that recovers domain-specific objectives and their weights under both controlled and real-world settings, explaining over 90% of reward behaviors quantitatively. It significantly outperforms baselines in detecting alignment faking cases. The project is advised by @sanmikoyejo, Carlos Guestrin, and others.
More from Safety
- Researcher quits Anthropic, says OpenAI and Anthropic are racing to self-improving superintelligence — ShakeelHashim · 2026-09-11
- Why So Many AI Researchers Think the Machines Could Kill Everyone — wiredmagazine · 2026-09-11
- California creates standards for independent AI auditors to verify lab safety testing — VraserX · 2026-09-11
- a16z podcast: why 2-3 person startups are absent from policy debates — a16z Podcast · 2026-09-11
- Researcher questions AI safety eval firm, citing 'blatantly sloppy' security and monitoring — Kyrannio · 2026-09-11
- Class action accuses Anthropic of overselling Claude subscriptions with deceptive usage multipliers — The Decoder · 2026-09-11