LLM gateway latency benchmarked: 0.21ms median overhead but 89ms at p99

jkriket · reddit · 2026-11-13

Aklivity's founder benchmarked Zilla, their open-source-based LLM gateway, to quantify the latency overhead introduced when routing AI agent traffic through a gateway.

Key findings:

On a 4-vCPU AWS instance, Zilla handled 150,000 requests successfully at 5,000 req/s
Median end-to-end latency was 1,502ms vs 1,500.9ms connecting directly to a mock provider; internal gateway processing was just 0.21ms median, with model routing at 20µs
At p99, however, added end-to-end latency was 89ms — a large gap between internal processing time and client-perceived latency
Streaming benchmark across OpenAI and Anthropic API dialects at 2,000 req/s: Zilla's p99 stream completion ranged 1.6-2.1s vs 2.0-9.3s for Bifrost

Caveats: synthetic benchmarks against a mock provider with fixed 1.5s responses, not full agent workflows, and streaming comparisons showed run-to-run variance. The takeaway: agents make repeated sequential calls, so tail latency and streaming behavior compound in complex workflows — centralized routing simplifies infrastructure, but the gateway must not become an unpredictable latency source. Full methodology and reproducible configurations are published.

Original post →

LLM gateway latency benchmarked: 0.21ms median overhead but 89ms at p99

More from coding & agent