Ramp SWE-Bench v2: top four coding models within points, but spend varies wildly
charliermarsh · x · 2026-11-19
Ramp's SWE-Bench v2 chart (pass@1, N=500, high effort) shows the top four coding models clustered within a few points (72-77%) even as mean cost per task ranges from $1 to $3+. Performance scales with model size and test-time compute but returns compress quickly. Spending patterns differ sharply: Gemini Carbon generates 4x the output tokens of Claude Opus 5.5 and GPT 6.1 Terra; Anthropic models use the fewest turns, OpenAI's are fastest. DeepSeek Harness's V4.1 Flash hits 61% at $0.15 per task, a standout on cost-efficiency.
Related event: Ramp Launches SWE-Bench v2 as Top Models Converge in Performance(2 posts)→
More from coding & agent
- Devs flag GPT's defensive programming for hiding bugs; OpenAI asks for /feedback cases — pvncher · 2026-11-19
- Flutter ships A2UI 1.0: agents respond with real interactive widgets, not chat text — rseroter · 2026-11-19
- Skills vs MCP vs RAG vs Memory: the four context mechanisms AI agents actually need — MaryamMiradi · 2026-11-19
- Anthropic details agent memory: 'dreaming' lets Claude organize its experience — daniel_mac8 · 2026-11-19
- The case for externalized memory as the real fix for continual learning — daniel_mac8 · 2026-11-19
- FT: US tech outages blamed on rush to fix bugs found by AI — nordicinst · 2026-11-19