Stripe launches coding-agent evals: agents meet all criteria in only 29% of runs
hazelcough · x · 2026-11-20
Stripe launched evals (stripe.dev/evals) measuring how well coding agents build real Stripe integrations. Key learnings:
- Stripe's plugin (recommended skills and tools) helped all 14 harness/model pairs tested, lifting scores 3-8 percentage points
- With the plugin, agents searched less and wrote fewer tests: web searches dropped 96% and extraneous test code fell 40% while scores improved
- Methodology uses another agent to role-play the developer, sharing imperfectly-stated goals and working through follow-up rounds
- The last mile is hard: agents satisfied 78% of criteria on average, but only 29% of runs met every required criterion
Charts compare Claude Code, Codex and OpenCode across one-time payments, subscriptions and Connect integrations.
Related event: Stripe Open-Sources Agent Evals: Only 29% of Coding Agents Fully Pass(6 posts)→
More from coding & agent
- Together AI webinar Dec 15: step-by-step RL post-training run on Kimi K3.1 — togethercompute · 2026-11-20
- GrayPass (YC F26) hits $66K MRR in two months policing AI agent actions in apps — ycombinator · 2026-11-20
- Google releases free 2-hour course covering full Harness engineering from agent to self-improving graph — udmrzn · 2026-11-20
- Zelda's Link runs Hermes Agent, and Teknium approves — Teknium · 2026-11-20
- Anthropic's Claude Platform OAuth lets agent platforms ditch API keys entirely — udmrzn · 2026-11-20
- Google's Antigravity in beta for 2 years still can't handle remote file uploads — jasondeanlee · 2026-11-20