ICML Paper: Reward Functions Are Only Partially Identifiable, Even With Infinite Data
gleech · x · 2026-09-07
gleech cites the ICML 2023 paper "Invariance in Policy Optimisation and Partial Identifiability in Reward Learning" (Skalse, Farrugia-Roberts, Russell, Abate, Gleave) as a pillar of the alignment thread: multiple reward functions often fit reward-learning data equally well, so the reward function is only partially identifiable even in the infinite-data limit. The paper formally characterizes partial identifiability across data sources like expert demonstrations and trajectory comparisons, and analyzes downstream impact on policy optimization — shifting the underlying true generator hurts objectives ("values") more than competence.
More from Safety
- Exponential launches RepoInject and BrowserInject benchmarks for prompt injection robustness — maksym_andr · 2026-10-27
- Poster surprised that v4.1 flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Redwood: CoT controllability evals are under-elicited; better prompts boost scores 2-3x — dfrsrchtwts · 2026-10-27
- Dev claims Anthropic is baking 'sabotage' into model behavior for 'wrong' engineering — cephaloform · 2026-10-27
- Anthropic IPO filing warns catastrophic AI incident could exceed entire insurance industry's capacity — TinfoilTricorn · 2026-10-27
- Researcher: assume all LLMs are situationally malicious — never run agents outside a VM — jd_pressman · 2026-10-27