Running Qwen3.8-Flash-Next on 2x3090: experts to RAM, 51B n-gram table on NVMe
jbro1985 · reddit · 2026-08-29
A Reddit user details deploying Qwen3.8-Flash-Next (125B MoE + 51B n-gram table) on 2x RTX 3090 + 96GB DDR5, with experts offloaded to RAM and the n-gram table mmap'd to NVMe.
Llama.cpp
- 32 t/s decode / 463 t/s prefill at UD-Q4KXL with a 640K serving pool (4x160K lanes, q8 KV)
- The 51B table shows 0% cross-token reuse, self-regulating to near-zero residency at zero decode cost — SSD random reads are the right architecture, not a compromise
- --moe-cache 5000 (PR #24528 branch): +30% decode; concurrency erases it — fan out shallow, serialize deep
- Warm prefix caching reprocesses only 24 tokens per turn (DeltaNet state included)
- Found and fixed a gridDim.y overflow crash (same shape as pending upstream PR #27941, 2-line fix)
- cgroup write-caging (MemoryMax=6G) keeps 32 t/s while pulling 90GB
vLLM
- Grafted unmerged mmap-PLE PR (#54129): table served from NVMe, W4A16, TP=2 — official recipe wants ≥110GB free RAM, this runs in 53GB
- Real-table decode 15.3 t/s vs 16.0 stubbed (4% NVMe cost)
- Selective experts-only UVA offload is mandatory: naive --cpu-offload-gb recopies everything each step (2.5 t/s); routing routedexperts weights gives 16 t/s
- MTP speculation hurts when PCIe is the bottleneck (13.9 vs 16.0)
Related event: Enthusiasts Test Local Qwen Flash Next: Memory Buys Quality but Speed Lags(2 posts)→
More from Infra
- Magnitude open-source inference engine beats Ollama by over 2x with auto-tuned kernels — nickbaumann_ · 2026-10-15
- Nvidia to announce $28B in AI investments in Europe this week, report says — georgemillo · 2026-10-15
- A $10,811 Cloudflare Bill From an Unused AI-Written Test Project: Refunds Aren't a Safety Net — sujingshen · 2026-10-15
- Retraining-Free Method Turns Transformers Into Elastic Models Adapting to Any Compute Budget — VISLab-Amsterdam · 2026-10-15
- vLLM's MTP3 validation: AIME25 up to 0.9667, prefill kernels up to 84% lower latency — vllm_project · 2026-10-15
- Open-weights s1-llm-auto-router picks the right LLM in 26ms on a CPU — airesearch12 · 2026-10-15