Stripe Open-Sources Agent Evals: Only 29% of Coding Agents Fully Pass
On November 20, Stripe released and open-sourced Stripe evals, a benchmark and evaluation framework for coding agents, along with initial results from a cross-comparison covering 14 model-harness combinations. Unlike conventional coding benchmarks that only check whether code passes deterministic tests, this benchmark verifies whether agents use the currently correct APIs and properly handle real integration issues such as payment failures and webhooks.
Confirmed
- The benchmark covers three scenarios — one-time payments, subscriptions, and Connect (marketplace/platform) — with 69 tasks total, evaluating 14 model-harness combinations
- Measured results: agents met 78% of criteria on average, but only 29% of runs met all criteria, indicating near-usability with frequent critical gaps
- Key issues: agents tend to use outdated APIs more common in their pretraining data, miss key features, and hallucinate
- Testing the official agent plugin: all 14 combinations improved by 3 to 8 percentage points; use of outdated API parameters dropped from 7% to 1.3%, and web searches fell 96% (agents instead relied on authoritative official guidance)
Why it matters
A large share of Stripe integration code is now written by coding agents, yet its correctness has long lacked targeted measurement. Stripe evals distinguishes "code that runs" from "correct API integration," offering the industry an evaluation paradigm oriented toward real-world business integration; the plugin's significant gains also show that giving agents authoritative documentation and tool guidance is more effective than letting them search on their own.
2026-11-20 ~ 2026-11-20 · 6 related posts
Primary sources
- Stripe open-sources coding-agent evals: 69 scenarios, agents fully pass only 29% of runs — emilygsands ·
- Stripe evals: coding agents favor outdated APIs and hallucinate — the plugin fixes much of it — emilygsands ·
- Stripe's agent plugin lifts all 14 coding agents by 3-8 points, cuts stale API usage from 7% to 1.3% — emilygsands ·
- Stripe open-sources evals benchmarking how well coding agents build real integrations — emilygsands · 2026-11-20
- [source] Stripe open-sources coding-agent evals: 69 scenarios, agents fully pass only 29% of runs — emilygsands · 2026-11-20
- [source] Stripe's agent plugin lifts all 14 coding agents by 3-8 points, cuts stale API usage from 7% to 1.3% — emilygsands · 2026-11-20
- [source] Stripe evals: coding agents favor outdated APIs and hallucinate — the plugin fixes much of it — emilygsands · 2026-11-20
- Stripe launches coding-agent evals: agents meet all criteria in only 29% of runs — hazelcough · 2026-11-20
1 near-duplicate retellings: emilygsands