3x 3090 runs 1M context: offload Qwen3.8-Flash-Next's KV cache to RAM with little slowdown
sadnessdevil · reddit · 2026-09-16
The author demonstrates that Qwen3.8-Flash-Next (built on qwen4exp) can keep most of its KV cache in system RAM with minimal decode slowdown, and patched vLLM to run 1M context on 3x RTX 3090.
Measured: 80 tok/s at short context, 60 tok/s once QSA hits its 2048-token budget, then flat as context grows; 150 tok/s at 4 concurrent requests; prefill 3,701 tok/s at 248k. Patches and model are on his Hugging Face page.
Why it works: decode is a bandwidth problem. A full-attention layer reads its entire KV cache every step — at 262k tokens that's 6GiB per step across 12 layers, and PCIe 4.0 x16 (32GiB/s) would allow only 5 tok/s. Flash-Next instead has only 12 of 48 layers with KV cache (the rest are fixed-size gated delta net layers), and QSA's indexer selects at most 2048 positions per layer: 48MiB per token total. Read volume is bounded by the indexer budget, not context length — 3.9GB/s sustains 80 tok/s. Only the weights plus 66B per token per layer must stay on GPU.
More from Infra
- Third study finds no significant pollution increase around xAI's Colossus 1 data center — chrisgrayson · 2026-09-16
- Anthropic inks first Australian data center deal: 2.16 GW campus near Brisbane, online 2027 — mark_k · 2026-09-16
- vLLM self-hosting guide: quantize an 8B model to quarter size with 98-100% accuracy — vllm_project · 2026-09-16
- Earendil launches Radius, an inference layer bringing tokens, routing and search to Pi — HankYeomans · 2026-09-16
- Multi-turn agent RL training at scale on HF Hub: 9,523 sandboxes in 14h, zero crashes — vanstriendaniel · 2026-09-16
- Apple Weighs M8-Based AI Server With Nvidia NVLink to Reenter Server Market — gappyvalley · 2026-09-16