InferenceBench: agents beat default inference engines but mostly lose to hyperparameter search
maksym_andr · x · 2026-11-09
InferenceBench's full findings: across four serving-bottleneck scenarios (prefill latency, decode latency, throughput, all-in-one), frontier coding agents outperform the vanilla PyTorch baseline and most inference engines with default configs (vLLM, SGLang, TGI), but — apart from the newest Claude Opus 5.5 (Max) run — remain worse than simple hyperparameter searches over existing engine settings given the same time budget. Each run gives the agent a 2-hour wall-clock budget to produce an OpenAI-compatible server; final submissions must pass correctness checks and an integrity audit for reward hacking, or score at baseline.
More from Models
- Meituan open-sources LongCat-2.5: 1.6T total params, ~48B active, MIT license — victormustar · 2026-11-09
- NetEase Youdao open-sources Confucius4-R2T2, a stable-prefix streaming ASR with 80ms chunks — pmttyji · 2026-11-09
- RL-trained harness adapters drop for LFM3, LFM2.5 and TwiL-8B to boost agentic coding — yb2698 · 2026-11-09
- AI eval chart dispute: is the model guessing what the grader wants to see? — teortaxesTex · 2026-11-09
- Google rebuilds Gemini API docs inside AI Studio, screenshot appears to show Gemini 4 Argon — patloeber · 2026-11-09
- Google AI Plus subscribers lose Gemini Pro model access starting November 13 — dungeoncrawler11 · 2026-11-09