Stripe open-sources coding-agent evals: 69 scenarios, agents fully pass only 29% of runs
emilygsands · x · 2026-11-20
Stripe open-sourced its coding-agent evals benchmark and harness: 69 scenarios across payments, subscriptions, and Connect checking real integration correctness (current APIs, failed payments, webhooks) across 14 coding agents. Agents meet on average 78% of criteria, but only 29% of runs satisfy every required criterion.
Related event: Stripe Open-Sources Agent Evals: Only 29% of Coding Agents Fully Pass(5 posts)→
More from coding & agent
- Stripe launches coding-agent evals: agents meet all criteria in only 29% of runs — hazelcough · 2026-11-20
- Google's Antigravity in beta for 2 years still can't handle remote file uploads — jasondeanlee · 2026-11-20
- Firecrawl launches Grok Bot plugin with 3,300+ data tools, claims 21% answer quality lift — devdigest · 2026-11-20
- OnchainBench launches: benchmarking whether crypto AI agents can be trusted with your keys — DavideCrapis · 2026-11-20
- Fig previews RoleCUA: 780+ hours of expert work to train agents for month-long job functions — adityaag · 2026-11-20
- Diagrams, Drawn by Agents: Using DrawingML to Make AI-Generated Charts Editable in PowerPoint — kristoph · 2026-11-20