Same Model, Different Backend: 8x Slower First Token
TypicalOcelot9 · reddit · 2026-07-17
This post compares the real-world performance of the same open-weight model across two OpenAI-compatible backends, concluding that identical API schemas don't guarantee identical inference behavior.
The author found that a dedicated inference host performed significantly better in cache-miss scenarios: TTFT was roughly 1.2s, throughput was 77 tok/s, and it genuinely streamed tokens incrementally. In contrast, a major cloud provider's hosted endpoint claimed to support stream:true but actually buffered the entire response, outputting it all at once after about 10s, causing tangible stalls in the agent loop.
Further investigation into network latency and local proxies confirmed the issue was server-side buffering. Crucially, under a real cache-hit distribution, that 10s delay (exacerbated by cache misses) shrinks significantly, dropping TTFT from 10s to 2.7s with minimal buffering. The post concludes with three benchmarking principles:
- OpenAI-compatible only dictates the request format, not behavioral quality
- stream:true might be superficial; always verify if the output is truly incremental
- Benchmarks must reflect real cache-hit distributions rather than relying solely on worst-case cache-busted scenarios
More from Infra
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11
- PyTorch Day Korea 2026 launches first offline conf, CFP closes Sept 13 — PyTorch · 2026-09-11
- Local LLM server dilemma: 4x CMP-170HX (price up 53% in 20 days) vs Mac Studio M5 Ultra — rumboll · 2026-09-11
- llama.cpp lands Flash Attention tuning for RDNA4, big prefill gains on AMD — pmttyji · 2026-09-11
- Your p99 latency benchmark may be lying: a deep dive into coordinated omission — Franc0Fernand0 · 2026-09-11