Warp Cut Cost per PR From $80 to $20 by Benchmarking Models on Its Own Traces
vikvang1 · x · 2026-10-30
A Warp engineer argues that 3D game demos aren't a good way to judge a model's coding ability — and neither are SWE-bench, CursorBench, or TermBench. Instead, benchmark models against your own traces to find the best fit for your tasks.
The Warp team built an internal benchmarking pipeline over various harness/model configurations in their software factory, cutting cost per PR from $80 to $20 and uncovering a number of unintuitive findings along the way.
Takeaway: choose models by evaluating on your real workflow traces, not public leaderboards — it saves money and surfaces differences leaderboards miss.
More from coding & agent
- auto-optimize makes bedtools intersect 3-5x faster with 20x less memory, output unchanged — jmschreiber91 · 2026-10-30
- Anthropic API Key Leaked; Agents Detected and Rotated It Within 15 Minutes — tylerjdunn · 2026-10-30
- Google engineers break down a product-style pipeline for shipping agent skills — rseroter · 2026-10-30
- Coding agents seem to have converged on the same PR format, sparking weight-training jokes — mattpocockuk · 2026-10-30
- Vercel Plugin Arrives in Google Antigravity's New Plugin Marketplace — steren · 2026-10-30
- Shopify execs require working prototypes in meetings, built with Codex — OpenAIDevs · 2026-10-30