AutoVend benchmark: 4 of 5 LLM agents lose to a rule-based baseline running 20 vending machines
Qiming Shi · hf · 2026-11-10
AutoVend is a benchmark for long-horizon autonomous business operations: one LLM agent manages 20 vending machines across five site types (factory, campus, residential, commercial, office) through a web interface for 180 simulated days.
- The data-grounded simulator includes a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs, testing whether online decisions translate into sustained economic returns.
- Four of five evaluated LLM agents fall short of a rule-based inventory-control baseline.
- Ranking models by returns per dollar of inference cost differs from ranking by returns alone.
- Trajectory analysis reveals persistent difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.
More from coding & agent
- Agents build Lean models of every CPython 3.15.2 function, validated on 50,000+ generated inputs — ChrSzegedy · 2026-11-10
- 10 open-source repos that review your code with AI, from Alibaba to Anthropic — victor_explore · 2026-11-10
- The dialectics of AI coding: LLMs can't write prod code unsupervised — neither can humans — ShenRaphael · 2026-11-10
- SteerCamp's multi-agent crew holds daily standups, reporting progress and blockers like employees — pramodk73 · 2026-11-10
- Running Codex for two days to reimplement an outdated Android book's projects and retake screenshots — burkov · 2026-11-10
- Unverified: Fable 5.2 tops FrontierCode 2.0 at 49.1%, half the cost of Opus — FinanceYF5 · 2026-11-10