Inference benchmark gotchas: bs=1-only engines are 'model runners,' warn practitioners
VoidAsuka · x · 2026-11-15
Responding to a SemiAnalysis investigation into inference benchmarks, practitioners warn of common pitfalls and overclaims: AL/AR (average length/accept length) must be controlled. SemiAnalysis says synthetic AL/AR was imposed to prevent finetuning draft models on benchmark prompts, though it remains a valid optimization for narrow use cases. As one engineer put it: an engine that only measures bs=1 isn't an inference engine, it's a model runner.
More from Infra
- Alphabet's free cash flow turns negative for first time in 20 years, burning ~$15B over two quarters — luisdans · 2026-11-15
- Reddit debate: Do US chip export controls slow down global open-source AI software too? — No-Birthday-7360 · 2026-11-15
- Revolut postmortem: 5-hour global cards outage caused by one GCP zone, no DB monitoring — cyb3rops · 2026-11-15
- China's state broadcaster claims all new electricity demand will be met by clean energy by 2030 — Polymarket · 2026-11-15
- ChatGPT verdict: NVIDIA's real product is market confidence, and a Lehman moment may be brewing — bingxu_ · 2026-11-15
- NVIDIA's 10-Q reveals billions in private-equity bets on KKR and Helix, drawing scrutiny — SumitGup · 2026-11-15