3x 3090 runs 1M context: offload Qwen3.8-Flash-Next's KV cache to RAM with little slowdown
sadnessdevil · reddit · 2026-09-16
The author demonstrates that Qwen3.8-Flash-Next (built on qwen4exp) can keep most of its KV cache in system RAM with minimal decode slowdown, and patched vLLM to run 1M context on 3x RTX 3090.
Measured: 80 tok/s at short context, 60 tok/s once QSA hits its 2048-token budget, then flat as context grows; 150 tok/s at 4 concurrent requests; prefill 3,701 tok/s at 248k. Patches and model are on his Hugging Face page.
Why it works: decode is a bandwidth problem. A full-attention layer reads its entire KV cache every step — at 262k tokens that's 6GiB per step across 12 layers, and PCIe 4.0 x16 (32GiB/s) would allow only 5 tok/s. Flash-Next instead has only 12 of 48 layers with KV cache (the rest are fixed-size gated delta net layers), and QSA's indexer selects at most 2048 positions per layer: 48MiB per token total. Read volume is bounded by the indexer budget, not context length — 3.9GB/s sustains 80 tok/s. Only the weights plus 66B per token per layer must stay on GPU.
More from Infra
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02
- Intel plunges 29%, worst day since the 1970s, wiping $12B off US government stake — emmanuelvivier · 2026-11-02
- Researcher hoards nearly 1TB of DRAM, refuses to sell a stick to data centers — kevinnbass · 2026-11-02
- Self-Hosting LLMs at a Law Firm: Infra and Data Governance Harder Than the Model — EducationalGroup7245 · 2026-11-02