Stripe evals: coding agents favor outdated APIs and hallucinate — the plugin fixes much of it
emilygsands · x · 2026-11-20
Stripe's blog details why it built Stripe evals and what it found:
- Problem: agents bias toward older APIs prevalent in pretraining corpora, miss key features, and hallucinate; existing coding benchmarks check deterministic test passes, not integration correctness
- Method: 69 realistic scenarios (hosted checkout, metered billing, per-seat subscriptions, marketplace splits) driven by a roleplaying agent, across 14 model-harness pairs, with and without the Stripe agent plugin
- Results: the plugin lifts all 14 pairs by 3-8 percentage points and cuts web searches 96% as agents rely on plugin-provided skills and Stripe MCP tools; models also behave differently with it
- Stripe will publish results periodically to help developers pick the most effective agent setup
Related event: Stripe Open-Sources Agent Evals: Only 29% of Coding Agents Fully Pass(5 posts)→
More from coding & agent
- Stripe launches coding-agent evals: agents meet all criteria in only 29% of runs — hazelcough · 2026-11-20
- Google's Antigravity in beta for 2 years still can't handle remote file uploads — jasondeanlee · 2026-11-20
- Firecrawl launches Grok Bot plugin with 3,300+ data tools, claims 21% answer quality lift — devdigest · 2026-11-20
- OnchainBench launches: benchmarking whether crypto AI agents can be trusted with your keys — DavideCrapis · 2026-11-20
- Stripe open-sources coding-agent evals: 69 scenarios, agents fully pass only 29% of runs — emilygsands · 2026-11-20
- Fig previews RoleCUA: 780+ hours of expert work to train agents for month-long job functions — adityaag · 2026-11-20