Accelerate Bench: 32 CPU optimization tasks separate 16 coding agents, top 2 hit 2x+ on 22 tasks
geoffwolfe · x · 2026-11-20
A new arXiv benchmark, Accelerate Bench, tests coding agents on application-level program optimization:
- 32 CPU optimization tasks adapted from scientific applications and supercomputing contests; each provides a working program plus workload, correctness requirements, timing bounds and permitted edits, leaving the agent to choose what to optimize. Public dev tests plus a separate verifier score final submissions.
- 16 model+harness combos scored 25.5%–67.4% on Score@1. Most programs were correct — speedup size was the differentiator: the top two agents hit 2x+ speedups on 22+ of 32 tasks, while the other 14 managed 2x on only 8–17 tasks.
- Controlled replays show why cross-application coordination matters: identical source ran between 0.29x and 2.12x depending solely on build/runtime settings, and one code change was worth 1.01x or 6.4x depending on another.
More from coding & agent
- TestSprite 4.0 launches as an independent test layer for AI-written code — Tegadesigns · 2026-11-20
- VP of engineering: coding models report like bad direct reports — raw dumps, no TLDR — robleclerc · 2026-11-20
- Lawyer runs full DeepSeek-V4.1-Flash on 12x V100 at 157-184 tok/s, beating DGX Spark stacks — TumbleweedNew6515 · 2026-11-20
- GitHub open-sources Agent Host Protocol for cross-device agent sessions — rseroter · 2026-11-20
- 100% code-driven Blender squirrel film, made for $34.76 — no video gen models — dreamwieber · 2026-11-20
- How Cisco upskilled 1,200 engineers in agentic AI via AWS AI League tournaments — AWS ML Blog · 2026-11-20