Stripe Open-Sources Agent Evals: Only 29% of Coding Agents Fully Pass

On November 20, Stripe released and open-sourced Stripe evals, a benchmark and evaluation framework for coding agents, along with initial results from a cross-comparison covering 14 model-harness combinations. Unlike conventional coding benchmarks that only check whether code passes deterministic tests, this benchmark verifies whether agents use the currently correct APIs and properly handle real integration issues such as payment failures and webhooks.

Confirmed

Why it matters

A large share of Stripe integration code is now written by coding agents, yet its correctness has long lacked targeted measurement. Stripe evals distinguishes "code that runs" from "correct API integration," offering the industry an evaluation paradigm oriented toward real-world business integration; the plugin's significant gains also show that giving agents authoritative documentation and tool guidance is more effective than letting them search on their own.

2026-11-20 ~ 2026-11-20 · 6 related posts

Primary sources

1 near-duplicate retellings: emilygsands