Ramp releases SWE-Bench v2: 500 production-built tasks, 6.2 Astra tops at 76.4%
AniBaddepudi · x · 2026-11-19
Fintech company Ramp published Ramp SWE-Bench v2, a coding-agent benchmark built from real engineering work:
- Tasks: 500 tasks derived from code shipped to production by 200+ Ramp engineers, with more rigorous verification
- Native harnesses: Frontier models evaluated in their native harnesses (pass@1, N=500, high effort, 95% CI)
- Leaderboard (chart): 6.2 Astra leads at 76.4%, Carbon 75.4%, Fable 5.2 and 6.2 Sol tied at 73.4%, Hydrogen 71.6%; Opus 5.5 scores 67.8%, Grok 4.7 64.4%, Sonnet 5.5 63.2%, with Ember last at 51.8%
Related event: Ramp Launches SWE-Bench v2 as Top Models Converge in Performance(2 posts)→
More from coding & agent
- Devs flag GPT's defensive programming for hiding bugs; OpenAI asks for /feedback cases — pvncher · 2026-11-19
- Flutter ships A2UI 1.0: agents respond with real interactive widgets, not chat text — rseroter · 2026-11-19
- Skills vs MCP vs RAG vs Memory: the four context mechanisms AI agents actually need — MaryamMiradi · 2026-11-19
- Anthropic details agent memory: 'dreaming' lets Claude organize its experience — daniel_mac8 · 2026-11-19
- The case for externalized memory as the real fix for continual learning — daniel_mac8 · 2026-11-19
- FT: US tech outages blamed on rush to fix bugs found by AI — nordicinst · 2026-11-19