InferenceBench: agents beat vLLM defaults but lose to simple hyperparameter search
maksym_andr · x · 2026-11-13
InferenceBench tests whether frontier coding agents can optimize LLM serving under a fixed compute budget. New results cover Fable 5.2, Sonnet 5.5, and Haiku 5.5.
- Across all four scenarios (prefill latency, decode latency, throughput, all-in-one), agents beat the vanilla PyTorch baseline and most inference engines with default configs (vLLM, SGLang, TGI).
- However, apart from Claude Opus 5.5 (Max) and Sonnet 5.5 (Max), agents remain worse than a simple hyperparameter search over existing engine settings given the same time budget.
- Fable 5.2 performs at the Fable 5.1 level and does not outperform previous Anthropic models, likely related to nerfs in AI R&D capabilities.
Setup: each run gives the agent a base model, hardware, and a two-hour budget to produce an OpenAI-compatible inference server, scored on speedup over the PyTorch baseline. Final submissions must pass correctness checks and a reward-hacking integrity audit, or are scored at baseline.
More from Models
- Uncensored LFM3 fork drops refusal rate from 61.7% to 0% in a 1.1GB file — alexcovo_eth · 2026-11-13
- Edge0-145B: an open-source 145B-parameter model claiming frontier capabilities on your own computer — BrightCandle · 2026-11-13
- Inkling 2 Small architecture detailed: attention residuals, NVFP4/MXFP8 mixed-precision experts — teortaxesTex · 2026-11-13
- Inkling 2 Small scores 31 on AA index, tops GLM and DeepSeek on finance evals — simonguozirui · 2026-11-13
- Gemini glitch leaks its backend response instructions to the user in plain text — SushiManPlatinum · 2026-11-13
- Aside's Computer Use agent hits SOTA on Scale AI's Cua Driver Bench — garrytan · 2026-11-13