Google engineers: LLM benchmark harnesses silently drop requests — 200 QPS in, 38 out
AI Engineer · youtube · 2026-09-20
Google engineers Ashok Chandrasekar and Jason Kramberger show why many published LLM inference benchmarks don't reproduce: Python GIL-bound harnesses asked for 200 QPS quietly delivered 38; busy clients inflated measured latency by up to 58 seconds; a "20% throughput gain" came from temperature 0; and two harnesses tokenized the same dataset differently. Their fix is Inference Perf, a CNCF multiprocess load generator with client-side telemetry vs server metrics, verified at 5,000 QPS, plus a declarative workload catalog and the Prism UI from llm-d.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04