Same Model, Different Backend: 8x Slower First Token
TypicalOcelot9 · reddit · 2026-07-17
This post compares the real-world performance of the same open-weight model across two OpenAI-compatible backends, concluding that identical API schemas don't guarantee identical inference behavior.
The author found that a dedicated inference host performed significantly better in cache-miss scenarios: TTFT was roughly 1.2s, throughput was 77 tok/s, and it genuinely streamed tokens incrementally. In contrast, a major cloud provider's hosted endpoint claimed to support stream:true but actually buffered the entire response, outputting it all at once after about 10s, causing tangible stalls in the agent loop.
Further investigation into network latency and local proxies confirmed the issue was server-side buffering. Crucially, under a real cache-hit distribution, that 10s delay (exacerbated by cache misses) shrinks significantly, dropping TTFT from 10s to 2.7s with minimal buffering. The post concludes with three benchmarking principles:
- OpenAI-compatible only dictates the request format, not behavioral quality
- stream:true might be superficial; always verify if the output is truly incremental
- Benchmarks must reflect real cache-hit distributions rather than relying solely on worst-case cache-busted scenarios
More from Infra
- Engineer with thousands of fabricated devices pours cold water on photonic crystal modulator yields — jwt0625 · 2026-10-26
- Anthropic S-1 reveals 48 cents of compute cost per dollar of Q2 revenue — nmasc_ · 2026-10-26
- Persistent AI joins UR network to route private AI inference with encrypted, IP-hidden traffic — 0xSammy · 2026-10-26
- Anthropic's disclosed compute commitments: $114.5B Google, $161.2B Broadcom top the list — BenBajarin · 2026-10-26
- SpaceX strikes $40B deal with Apollo Global to lease AI chips and data center hardware — Polymarket · 2026-10-26
- SpaceX AI revenue up 830%, Claude designs 318K antivenom proteins in 64 hours — PeterDiamandis · 2026-10-26