MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them
TheKanter · x · 2026-10-16
MLCommons introduced MLPerf Endpoints, a benchmark that measures a full served inference endpoint—model, hardware, serving software, and networking together—under production-like load, instead of an isolated tuned configuration.
Key points:
- Full operating frontier, not a point: results are a Pareto frontier across concurrency levels, so buyers can match throughput and responsiveness to their own workloads and SLAs
- Five primary metrics: concurrency, Time to First Token P90, system throughput (tokens/s), throughput per watt (a TCO proxy), and per-user interactivity
- Rolling submissions: publication cohorts every few weeks keep results current with models and software
- Peer-reviewed status levels: verified, provisional, or unverified
Related event: MLCommons launches MLPerf Endpoints, picked for EU AI Gigafactories(2 posts)→
More from Infra
- Meme: driving to the chip factory in 2028 captures AI's compute anxiety — beffjezos · 2026-10-16
- One volunteer maintains the time zone file behind 4 billion phones — danbri · 2026-10-16
- DeepSeek V4.1 Flash on 4x DGX Spark: weights alone are 511GB, the 494 tok/s claim doesn't fit — Aroochacha · 2026-10-16
- Open-sourced Intel GPU Vulkan inference upgrades: 2x ingestion, 60%+ faster inference — Fun-Aardvark-1143 · 2026-10-16
- ELSFP pain points: >3dB insertion loss, costly PMF, harsh packaging requirements — jwt0625 · 2026-10-16
- Neocloud M&A decoded: CoreWeave, Nebius, Crusoe, Nscale all bet margins vanish beyond GPU scarcity — AccBalanced · 2026-10-16