65x speedup: 40,000-word output drops from 62.7s to 961ms
DanielLockyer · x · 2026-10-19
The author shared a before/after benchmark table for chunked output: 2,000 words went from 180ms to 31ms (6×), 8,000 words from 2.4s to 110ms (22×), 20,000 words from 15.3s to 292ms (52×), and 40,000 words (52,308 chunks) from 62.7s to just 961ms — a 65× speedup. Gains scale with output length, indicating the optimization targets cumulative latency in long-form streaming output.
More from Infra
- llama.cpp changes default server port from 8080 to 9931 in v0.7.0 — 413205 · 2026-10-19
- Microsoft pulls out of £2B UK AI data center as grid delays push power into 2030s — Polymarket · 2026-10-19
- Local Qwen3 27B generates a full video in ~2 hours, completely free — aziz4ai · 2026-10-19
- Poliscope Made Germany's 4,800 Council Portals Searchable on One Qdrant Server, Cutting Latency from 20s to 0.87s — qdrant_engine · 2026-10-19
- GPUs literally falling off the bus: MCIO riser triggers NVIDIA Xid 79 errors — TheZachMueller · 2026-10-19
- 763B DeepSeek-V4.1-Flash runs locally at 500+ tok/s on 4x RTX Pro 6000 — TheZachMueller · 2026-10-19