KV cache offload to RAM runs Qwen 3.8 with 8x200k streams on 4x3090

Mr_Moonsilver · reddit · 2026-11-23

Reddit user MrMoonsilver released an experimental vLLM overlay (crazydiamond-on-ampere) running Intel/Qwen3.8-Flash-Next-W4A16-AutoRound on 4x3090 with 256GB DDR4. Weights stay fully in GPUs while KV cache is offloaded to system RAM; 2TPx2PP with NVLink pairs yields 8 concurrent streams of 200k context each at 480 tk/s total and 6.2k prefill. A fifth 3090 hosts the vision encoder. No MTP yet; the author plans an updated version and invites community iteration.

Original post →

More from coding & agent

coding & agent channel →