LLM gateway benchmarks: median latency tiny, but p99 and concurrent streaming hurt
Benchmarks of an open-source LLM gateway showed median added latency of just 0.21ms but p99 spikes of 89ms, while practitioners warn that advertised microsecond figures obscure the real bottleneck: concurrent streaming performance.
2026-11-13 ~ 2026-11-13 · 2 related posts
- Benchmarking LLM gateways: p50 latency hides the real bottlenecks in streaming under load — jkriket · 2026-11-13
- LLM gateway latency benchmarked: 0.21ms median overhead but 89ms at p99 — jkriket · 2026-11-13