Revisiting Distill's 'Understanding RL Vision': attribution-based interpretability tricks that still matter
JacobHHilton · x · 2026-10-16
- Jacob Hilton revisits the 2020 Distill article Understanding RL Vision, co-authored with Nick Cammarata, Shan Carter, Gabriel Goh and Chris Olah at OpenAI, arguing most of its core difficulties remain wide open.
- The article applies interpretability to an RL agent on CoinRun: attribution combined with NMF dimensionality reduction builds an interface to inspect which in-game objects (coins, enemies, buzzsaws) the model detects and how they shape its value function and policy.
- Three showcase uses: dissecting failure (step-by-step analysis of missed max-reward episodes, e.g. an obstacle briefly obscured from view), hallucinations (features the model "hallucinates" in observations), and model editing (hand-editing weights to blind the agent to specific hazards, quantitatively validated).
More from Research
- SERO augments long-horizon agentic RL with executable oracle continuations — Shanghai-AI-Laboratory · 2026-10-16
- Tsinghua and Amap propose OmniVerifier-C1, a graded verifier beating binary judges — Bowei Liu · 2026-10-16
- Alibaba open-sources MAI-Gym, an all-in-one GUI agent eval and RL training platform — Tongyi-MAI · 2026-10-16
- Peking University's SimForcing distills simulation motion priors into robot world models — PekingUniversity · 2026-10-16
- CoWorld Reasoner uses an evolving video world model as a visual reasoning engine — ModalityDance · 2026-10-16
- Sparse rank-1 maps are often interpretable: a finding said to jump your neural net understanding 5 years ahead — gleech · 2026-10-16