ICML Study: Measuring Agents in Production Reveals Reliance on Simple Approaches
matei_zaharia · x · 2026-08-22
The paper Measuring Agents in Production (MAP), accepted at ICML 2026, presents the first systematic study of how LLM-based agents are deployed in real-world environments. Based on a survey of 86 practitioners and 20 in-depth case studies, the research highlights key findings:
Current State of Deployment
- Simplicity Prevails: 68% of agents involve human intervention within 10 steps; 70% rely on prompting off-the-shelf models instead of fine-tuning; 74% depend primarily on human evaluation.
- Top Challenge: Reliability (consistent correct behavior over time) remains the primary hurdle, currently addressed mostly through systems-level design.
Impact
MAP documents the reality of production agents, bridging the gap between academic benchmarks and industrial practices, while identifying under-explored avenues for future research.
Related event: ICML Paper Surveys 86 Production LLM Agent Systems(2 posts)→
More from coding & agent
- Leerob's Stanford Lecture Breaks Down Always-On Agents, Harnesses and Context Engineering — dean_rie · 2026-10-16
- Dev Praises Codex Local Computer Use: 'Every Other Harness Disappoints Me' — craigsdennis · 2026-10-16
- Decision models match LLMs on routing and safety — and are 7-8x faster — randal_olson · 2026-10-16
- Temporal Serverless Workers on Cloud Run: queue-driven autoscaling, scale to zero — rseroter · 2026-10-16
- Meta's Superintelligence Lab Runs 200+ Direct Reports—A Lesson for Flat Multi-Agent Systems — peterjliu · 2026-10-16
- LangChain CEO: the most interesting agent harness work lives at the harness-infra boundary — hwchase17 · 2026-10-16