Stripe open-sources coding-agent evals: 69 scenarios, agents fully pass only 29% of runs

emilygsands · x · 2026-11-20

Stripe open-sourced its coding-agent evals benchmark and harness: 69 scenarios across payments, subscriptions, and Connect checking real integration correctness (current APIs, failed payments, webhooks) across 14 coding agents. Agents meet on average 78% of criteria, but only 29% of runs satisfy every required criterion.

Related event: Stripe Open-Sources Agent Evals: Only 29% of Coding Agents Fully Pass(5 posts)→

Original post →

More from coding & agent

coding & agent channel →