Ramp Ships SWE-Bench v2 With 500 Real-World Tasks Testing Agent Code Merge-ability
charliermarsh · x · 2026-11-19
Ramp released Ramp SWE-Bench v2, a coding benchmark designed around real-world merge-ability of agent-written code:
- 500 tasks curated from actual PRs contributed by 200+ Ramp engineers, each requiring substantial changes to the initial agent implementation;
- Focuses precisely on the gap between what agents produce and what actually gets merged, across multiple domains;
- More rigorous verification, with frontier models run in their native harnesses;
- Ramp uses these benchmarks to allocate inference spend across providers, often shifting multiple times a week — "effectively all code is agent code."
Related event: Ramp Launches SWE-Bench v2 with 500 Real Production Tasks(3 posts)→
More from coding & agent
- Using /grill-me-with docs to grill requirements before building a Pokopia feature — feregri_no · 2026-11-19
- LangChain hosts Nov 19 live workshop on building agents with LangSmith Managed Deep Agents — LangChain · 2026-11-19
- ØMEGA Modus: a 7KB Markdown rule set that stops coding agents from faking "verified" work — MichaelS1011 · 2026-11-19
- OpenAI CLI Adds `explore`: Find Commands by Describing Them in Plain English — OpenAIDevs · 2026-11-19
- Mythical Creature: A Design Studio That Designs Directly in Your Codebase, No Figma — round · 2026-11-19
- a16z Leads Series B for Linq, Betting on Communication Infrastructure for AI Agents — Scobleizer · 2026-11-19