InferenceBench ranks agents on LLM serving speedup: Claude Opus 5.5 tops at 12.08x, agents lose to simple hyperparameter search
maksym_andr · x · 2026-11-21
InferenceBench tests whether frontier coding agents can optimize LLM serving under a fixed budget: each run gives an agent a base model, hardware, and a two-hour wall-clock budget to build an OpenAI-compatible inference server, scored by speedup over a PyTorch baseline across prefill latency, decode latency, throughput, and all-in-one scenarios, with correctness and reward-hacking integrity audits on final submissions.
Leaderboard: Claude Opus 5.5 (Max) leads at 12.08x geometric-mean speedup, Claude Sonnet 5.5 (Max) follows at 12.06x, Gemini 4 Neon (High) takes third at 10.96x — slightly ahead of GPT-6.2 Astra (Ultra) at 10.76x.
Key caveat: while agents beat the PyTorch baseline and most default-configured engines (vLLM, SGLang, TGI), all runs except the newest Claude Opus/Sonnet 5.5 (Max) underperform a simple hyperparameter search over existing engine settings given the same time budget.
More from coding & agent
- Mnemos 2.0: agent memory designed to preserve what matters for the next session — RileyRalmuto · 2026-11-21
- 4 architecture patterns to tame multi-agent system reliability, from a 27-year reliability engineer — bibryam · 2026-11-21
- AutoCut + FrameAtlas: open-source local-first video editing driven by Claude Code via MCP — moheshmohan · 2026-11-21
- InfoQ: What an agent harness is, and two ways to build one — bibryam · 2026-11-21
- The hidden 70 minutes: a Reddit cost breakdown shows agent workflows can cost more human time than manual — AlarmingPepper9193 · 2026-11-21
- Developer builds a real-time voice-and-screen coding agent with Claude for JARVIS-style pairing — Different_Scholar_74 · 2026-11-21