Mean scores mislead RL progress; Google Research proposes IQM and confidence intervals
burkov · x · 2026-08-26
Deep reinforcement learning experiments are expensive, leading researchers to compare algorithms using few runs despite high inter-run variance. A paper from Google Research and Université de Montréal uses large-scale Atari experiments to demonstrate how ordinary mean or median scores give a misleading picture of progress. The authors propose practical reporting methods for budget-constrained scenarios: confidence intervals, performance profiles, and the interquartile mean (IQM).
More from Research
- Schmidt Sciences awards $10M to 19 AI humanities research teams — OdedRechavi · 2026-10-15
- Mistral and CERN partner to build a Large Physics Model trained on billions of simulated collisions — MistralAI · 2026-10-15
- Global Transport coupling lifts guided flow generation: ImageNet FID 2.06 to 1.91 — mmbronstein · 2026-10-15
- Your breathing gives you away: Wi-Fi signals can reveal your breath rhythm — TinfoilTricorn · 2026-10-15
- Beyond text correlation: attaching a geometry simulator to ground model reasoning — gerardsans · 2026-10-15
- Sequen CEO Zoe Weil: Ranking Models Should Optimize Purchases, Not Clicks — TWIML AI Podcast · 2026-10-15