ICML oral: first systematic study of 86 production agent systems finds simplicity wins
berkeley_ai · x · 2026-08-22
The ICML 2026 oral talk "Measuring Agents in Production (MAP)" is now on video — the first systematic study of production LLM agents based on first-hand data:
- Method: 20 in-depth case studies plus a survey of practitioners across 86 deployed systems in 26 domains, covering why and how organizations build agents, how they evaluate them, and top challenges.
- Key findings: production agents favor simple, controllable approaches — 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation.
- Reliability (consistent correct behavior over time) is the top challenge, currently addressed through systems-level design.
Authors include Melissa Pan, Dawn Song, Ion Stoica, and Matei Zaharia, giving the research community visibility into deployment realities and underexplored avenues.
Related event: ICML Paper Surveys 86 Production LLM Agent Systems(2 posts)→
More from coding & agent
- Perplexity open-sources SPACE, the production sandbox platform behind Perplexity Computer — AravSrinivas · 2026-10-16
- Meta releases Muse Spark Open: 816B MoE, 42B active, 1M context, day-0 on Baseten — baseten · 2026-10-16
- Perplexity open-sources SPACE, the sandbox platform powering its Computer agents — AravSrinivas · 2026-10-16
- Vercel makes Python generally available: FastAPI, Django, Fluid Compute support — steren · 2026-10-16
- Full course: build a playable Unity fishing game with Claude Opus 5.5 in six steps — Roger_M_Taylor · 2026-10-16
- Marc Lou praises PostShiba, a cute indie alternative to AWS SES for email infra — marclou · 2026-10-16