Stripe's agent plugin lifts all 14 coding agents by 3-8 points, cuts stale API usage from 7% to 1.3%

emilygsands · x · 2026-11-20

Stripe shared results from its agent plugin evals: all 14 model-harness pairs improved by roughly 3-8 percentage points with the plugin, outdated API parameter usage dropped from 7% to 1.3%, and web searches fell 96% as agents relied on trusted guidance.

Baseline finding: the average agent meets 78% of evaluation criteria, but only 29% of runs satisfy every required criterion — agents get close but rarely all the way there.

Related event: Stripe Open-Sources Agent Evals: Only 29% of Coding Agents Fully Pass(5 posts)→

Original post →

More from coding & agent

coding & agent channel →