125B Qwen3.8-Flash-Next hits ~20 tok/s on 16GB VRAM via direct NVMe SSD offloading
MaxDev0 · reddit · 2026-10-25
Reddit user MaxDev0 ported sybil-solutions' efficient NVMe offloading approach (originally built for GLM-5.3) to the lighter Qwen3.8-Flash-Next (125B), avoiding the usual requirement of pinning all 24k experts in 48–64GB of system RAM.
Benchmarks (Q4 KV cache):
- RTX 4090 with only 8GB system RAM allocated: 28–33 tok/s prose / 19–22 tok/s code at 8k context (1,080 tok/s prefill); 20.8 tok/s decode at 100k; 20.0 tok/s decode at the full 262k window with 1,627 tok/s prefill
- RTX 3080 Laptop (16GB VRAM, 32GB RAM): 18–21 tok/s decode at 100k context, peak VRAM 15.5GB
Setup instructions, model downloaders, and packing scripts are open-sourced on GitHub (lean-qwenflash), built on ExLlamaV3 — a directly reproducible recipe for running large MoE models locally on modest hardware.
More from Infra
- OpenRouter analysis finds evidence of spoofing-style price manipulation by inference providers — AccBalanced · 2026-10-25
- 106 Tesla Megapacks to replace a former Belgian coal plant with 400 MWh storage — XFreeze · 2026-10-25
- Nebius cached-token pricing sends its OpenRouter traffic vertical, from under 20B to 67B tokens a day — AccBalanced · 2026-10-25
- Best setup to run Qwen3.8 27B locally on an M4 Pro 48GB Mac? — 9kSs · 2026-10-25
- Security researchers find 32 NVIDIA B300 GPUs on Chinese networks in scan of 12,000+ exposed GPUs — AccBalanced · 2026-10-25
- Xinjiang emerges as an AI compute hub with half-price power and 16.2ms latency to Chongqing — 96Stats · 2026-10-25