ICML oral: first systematic study of 86 production agent systems finds simplicity wins
berkeley_ai · x · 2026-08-22
The ICML 2026 oral talk "Measuring Agents in Production (MAP)" is now on video — the first systematic study of production LLM agents based on first-hand data:
- Method: 20 in-depth case studies plus a survey of practitioners across 86 deployed systems in 26 domains, covering why and how organizations build agents, how they evaluate them, and top challenges.
- Key findings: production agents favor simple, controllable approaches — 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation.
- Reliability (consistent correct behavior over time) is the top challenge, currently addressed through systems-level design.
Authors include Melissa Pan, Dawn Song, Ion Stoica, and Matei Zaharia, giving the research community visibility into deployment realities and underexplored avenues.
Related event: ICML Paper Surveys 86 Production LLM Agent Systems(2 posts)→
More from coding & agent
- Before building high-stakes AI systems: tell the LLM not to build anything — aryanXmahajan · 2026-08-22
- AI Refactoring Risks: Accidentally Deleting Valuable Technical Debt — sebpaquet · 2026-08-22
- MiniMax M3 ran overnight and found a TypeScript compiler crash—fuzzing first, then static analysis — DanielLockyer · 2026-08-22
- 10 Claude Code Steering Mechanisms That Stop Agents From Ignoring Instructions — bibryam · 2026-08-22
- DeepMind's Aletheia Agent Solves Math Conjectures via Self-Verification — thisguyknowsai · 2026-08-22
- Designing the Backend for Agent Systems: API, Queues, and Cost Control — kmeanskaran · 2026-08-22