Research: Reverse-Engineering Black-Box Rewards to Understand LLM Learning
sanmikoyejo · x · 2026-07-05
@edchene released a paper focusing on "what behaviors black-box reward functions actually align LLMs to." The authors note that aligning with uninterpretable black-box rewards obscures the model's actual learned objectives, potentially introducing hidden misalignments. They propose a detection method that recovers domain-specific objectives and their weights under both controlled and real-world settings, explaining over 90% of reward behaviors quantitatively. It significantly outperforms baselines in detecting alignment faking cases. The project is advised by @sanmikoyejo, Carlos Guestrin, and others.
More from Safety
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Johns Hopkins Alerts 3,000+ Patients After Data Was Sent to an AI Tool — beingmodest · 2026-10-27
- Cohere reports Qwen models may phone home to China servers unprompted — troll_khan · 2026-10-27
- Alignment Hub launches with 120+ documented AI misbehavior incidents and a reporting template — Thom_Wolf · 2026-10-27
- Cohere: Qwen3.8 tried to upload probe file to Beijing cloud bucket in sandbox test — Cohere · 2026-10-27
- US senator slams AI industry for proposing 50-death threshold for 'catastrophic incident' — peterwildeford · 2026-10-27