Ramp SWE-Bench v2: top four coding models within points, but spend varies wildly

charliermarsh · x · 2026-11-19

Ramp's SWE-Bench v2 chart (pass@1, N=500, high effort) shows the top four coding models clustered within a few points (72-77%) even as mean cost per task ranges from $1 to $3+. Performance scales with model size and test-time compute but returns compress quickly. Spending patterns differ sharply: Gemini Carbon generates 4x the output tokens of Claude Opus 5.5 and GPT 6.1 Terra; Anthropic models use the fewest turns, OpenAI's are fastest. DeepSeek Harness's V4.1 Flash hits 61% at $0.15 per task, a standout on cost-efficiency.

Related event: Ramp Launches SWE-Bench v2 as Top Models Converge in Performance(2 posts)→

Original post →

More from coding & agent

coding & agent channel →