KV cache offload to RAM runs Qwen 3.8 with 8x200k streams on 4x3090
Mr_Moonsilver · reddit · 2026-11-23
Reddit user MrMoonsilver released an experimental vLLM overlay (crazydiamond-on-ampere) running Intel/Qwen3.8-Flash-Next-W4A16-AutoRound on 4x3090 with 256GB DDR4. Weights stay fully in GPUs while KV cache is offloaded to system RAM; 2TPx2PP with NVLink pairs yields 8 concurrent streams of 200k context each at 480 tk/s total and 6.2k prefill. A fifth 3090 hosts the vision encoder. No MTP yet; the author plans an updated version and invites community iteration.
More from coding & agent
- Exploring LLMs + symbolic execution with Z3 for reverse engineering ideas — mgostIH · 2026-11-23
- Chat with Work launches tomorrow: cross-tool agent bots, EU-available and self-hostable — kieranklaassen · 2026-11-23
- Allie Miller's 'Fix It Days': batch-clearing household glitches one by one with Opus 5.5 — alliekmiller · 2026-11-23
- Replit agents speed up NumPy 18.9x — but only Claude's team scales — amasad · 2026-11-23
- A better yardstick for 10x engineers: how much of their work gets reused — vboykis · 2026-11-23
- Microsoft rebrands Azure AI Search as Foundry IQ, a knowledge layer for agents — adnan_hashmi · 2026-11-23