AI Agents That Matter
Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, Arvind Narayanan
cs.LG, cs.AI
2024-07-02
On HumanEval, GPT-4 warming retry hits 93.2% for $2.45; LATS is less accurate and ~50× costlier. Boards that ignore cost reward spend.
Agent leaderboards mostly report accuracy. Language models are stochastic, so extra samples and retries against unit tests raise the score. AlphaCode goes from near-zero zero-shot to over 15% with 1,000 samples and over 30% with a million. SWE-Agent capped each task at $4. If the board ignores the bill, the winning strategy is unbounded inference.
Kapoor, Stroebl, Narayanan, and colleagues at Princeton are not submitting another high-scoring agent. They argue that then-current evaluation habits mislead: complex scaffolds get credit for gains that cheap retries already buy, holdouts are missing so shortcuts work, and reproduction scripts diverge.
They put three public HumanEval "SOTA" agents on one ledger: LDB, LATS, and Reflexion, all of which debug, search, or "reflect" after example tests fail. Three almost architecture-free baselines sit next to them. Retry: temperature 0, up to five attempts (even temperature 0 is not fully deterministic). Warming: the same, with temperature ramped from 0 to 0.5. Escalation: start at Llama-3 8B, then GPT-3.5, Llama-3 70B, GPT-4 on failure. Each system is run five times on 164 problems; they report mean accuracy and USD cost at April 2024 prices.
A second thread modifies DSPy to jointly maximize HotPotQA accuracy and minimize few-shot tokens. A third uses NovelQA to split model evaluation from downstream procurement evaluation. A fourth inventories 17 agent benchmarks against the holdout their claimed generality actually requires. A fifth reproduces WebArena and HumanEval eval scripts and lists why numbers do not come back.
The HumanEval Pareto plot is the paper. Warming (GPT-4) averages 93.2% (92.1–93.9) at $2.45. Retry is 92.0% at $2.51. Zero-shot GPT-4 is already 89.6% at $1.93. LDB (GPT-4) hits 93.3% at $6.36: no accuracy edge, more than double the cost. Reflexion is 87.8% at $3.90. LATS (GPT-4) is 88.0% at $134.50, over fifty times Warming, and less accurate. Escalation reaches 85.0% for $0.27, cheaper and better than LDB (GPT-3.5) at $0.63. The original Reflexion 91% claim does not show up in five reproduction runs.
On HotPotQA, default DSPy lifts accuracy with up to eight few-shot examples and lifts variable cost with them. Joint optimization cuts variable cost 53% for GPT-3.5 and 41% for Llama-3-70B at matched accuracy. After about 1,350 tasks the one-time tuning cost is paid back versus default DSPy.
NovelQA concatenates a whole novel with every question, which is fine for long-context model tests and wrong for a product where users ask one question at a time. Sequential real use makes RAG about 21.9× cheaper than stuffing the book; the benchmark compresses that gap to about 2× and flatters long context.
STeP, then top of WebArena at 35.8%, more than doubles the original baseline. Its policies are hardcoded: a Reddit profile is the current host plus /user/username. A URL change breaks it. Of 17 benchmarks, appropriate holdouts are 3/6 task-specific, 1/8 domain-general, 0/2 fully general; seven have none and no plan to add one. HumanEval originally omits example tests on 3 of 164 tasks; agent papers each patch a different subset, then PapersWithCode folds them into one board.
In 2024 this reads as an evaluation constitution, not a new method. The failure modes it names keep recurring: boards that reward token burn; "reflection / debug / search" gains that collapse to extra samples; cost proxies (active parameters, training FLOPs) that mislead buyers; tiny benchmarks that let you compile the task into the policy. Downstream, the honest plot is accuracy versus dollars, with input/output token counts so later prices can be recomputed. For benchmark authors, holdout granularity has to match the generality you advertise.
HumanEval is easy. The authors say System 2 methods may still matter on SWE-bench-scale work. The paper does not kill planning or reflection. It kills declaring an architectural win without a cost control and without a retry baseline.
The reproductions concentrate on HumanEval plus thin slices of WebArena, NovelQA, and HotPotQA. Tool-heavy, long-horizon environments are a longer step. Dollar figures use April 2024 API list prices; absolute USD will rot, the 50× LATS gap is the durable bit. Warming ties SOTA partly because HumanEval ships an executable oracle that real software work rarely has. STeP's authors wanted composable policies for known tasks, which is a different goal from "realistic web agents," but the leaderboard number still gets read as general skill. Human-in-the-loop is almost absent; other work shows simple feedback lifting GPT-4 from 0% to over 86% on hard programming, so fully automatic scores both understate usefulness and overstate autonomy.