Same Model, Different Backend: 8x Slower First Token
TypicalOcelot9 · reddit · 2026-07-17
This post compares the real-world performance of the same open-weight model across two OpenAI-compatible backends, concluding that identical API schemas don't guarantee identical inference behavior.
The author found that a dedicated inference host performed significantly better in cache-miss scenarios: TTFT was roughly 1.2s, throughput was 77 tok/s, and it genuinely streamed tokens incrementally. In contrast, a major cloud provider's hosted endpoint claimed to support stream:true but actually buffered the entire response, outputting it all at once after about 10s, causing tangible stalls in the agent loop.
Further investigation into network latency and local proxies confirmed the issue was server-side buffering. Crucially, under a real cache-hit distribution, that 10s delay (exacerbated by cache misses) shrinks significantly, dropping TTFT from 10s to 2.7s with minimal buffering. The post concludes with three benchmarking principles:
- OpenAI-compatible only dictates the request format, not behavioral quality
- stream:true might be superficial; always verify if the output is truly incremental
- Benchmarks must reflect real cache-hit distributions rather than relying solely on worst-case cache-busted scenarios
More from Infra
- Why a 1GW Chinese AI data center may be plausible after all — teortaxesTex · 2026-07-22
- China’s AI arms race is increasingly defined by chips, data centers, and open models — BenBajarin · 2026-07-22
- Agent search bottlenecks are now about variance, not raw latency — rohanpaul_ai · 2026-07-22
- Gavin Baker argues Nvidia may be one of open source AI’s biggest supporters — GavinSBaker · 2026-07-22
- AI Power Demand Exposes US Energy Gap, Urging Shift from Scarcity to Abundance — bradneuberg · 2026-07-22
- Gavin Baker says Nvidia’s $630B figure would be system revenue, not all Nvidia’s — GavinSBaker · 2026-07-22