Benchmarking LLM gateways: p50 latency hides the real bottlenecks in streaming under load
jkriket · reddit · 2026-11-13
An engineer working on LLM gateway infrastructure shares hard-won benchmarking lessons. Headline 'microsecond overhead' numbers are misleading: routing logic, time inside the gateway, and client-perceived added latency are three different numbers. Practical advice: (1) benchmark against a direct baseline using a mock provider with fixed latency to isolate gateway/network/load-generator overhead; (2) use open-loop load generation, otherwise a slow gateway also slows request arrival and masks problems under load; (3) watch HTTP versions — at 5,000 req/s with 1.5s upstream responses that's 7,500 concurrent requests, and HTTP/1.1 vs HTTP/2 differs enormously; (4) benchmark streaming separately, since gateways fine with JSON can behave very differently with thousands of concurrent SSE streams, especially translating between OpenAI and Anthropic formats; (5) look beyond the median — p99 time-to-first-token and stream completion latency reveal bottlenecks averages hide. Conclusion: sustained-concurrency streaming behavior is a more meaningful production benchmark than routing microseconds.
More from Infra
- Meme explains CUDA perfectly: optimized for many Alice Huangs, not one — djcows · 2026-11-13
- Perplexity's Mac app shows a local models option, hinting at offline capability — dnlkwk · 2026-11-13
- 2-bit DeepSeek V4.1 Flash runs a coding-agent seat on two 2016 V100s: a 12-hour real-work trial — a4mula · 2026-11-13
- IBM QDC 2026 Unveils Nighthawk r2 Quantum Chip and Fault-Tolerant Dev Tooling — MJBiercuk · 2026-11-13
- Why Jensen Huang Splits From US Frontier Labs on Open Source and Export Controls — morganlai · 2026-11-13
- SGLang-Neuron v0.1.0 open-sourced: SGLang serving stack now runs on AWS Trainium — ying11231 · 2026-11-13