ICML Study: Measuring Agents in Production Reveals Reliance on Simple Approaches
matei_zaharia · x · 2026-08-22
The paper Measuring Agents in Production (MAP), accepted at ICML 2026, presents the first systematic study of how LLM-based agents are deployed in real-world environments. Based on a survey of 86 practitioners and 20 in-depth case studies, the research highlights key findings:
Current State of Deployment
- Simplicity Prevails: 68% of agents involve human intervention within 10 steps; 70% rely on prompting off-the-shelf models instead of fine-tuning; 74% depend primarily on human evaluation.
- Top Challenge: Reliability (consistent correct behavior over time) remains the primary hurdle, currently addressed mostly through systems-level design.
Impact
MAP documents the reality of production agents, bridging the gap between academic benchmarks and industrial practices, while identifying under-explored avenues for future research.
More from coding & agent
- Mod a phone stand and mic for the perfect voice input workstation — willcb · 2026-08-22
- Feature Flags for Agent Skills: A Production Cheatsheet — blaizedsouza · 2026-08-22
- Mixed-mode programming prototype enables natural language in REPL — deliprao · 2026-08-22
- GraphJin: Governed AI agent framework with zero unsafe effects — dosco · 2026-08-22
- Human-Agent Collaboration Accelerates Science: AI Redesigns Jet Engine Parts — enjalot · 2026-08-22
- Automate Weekly SEO/AEO Research with AI Agents for Free Traffic — RealGeneKim · 2026-08-22