Running Qwen3.8-Flash-Next on 2x3090: experts to RAM, 51B n-gram table on NVMe
jbro1985 · reddit · 2026-08-29
A Reddit user details deploying Qwen3.8-Flash-Next (125B MoE + 51B n-gram table) on 2x RTX 3090 + 96GB DDR5, with experts offloaded to RAM and the n-gram table mmap'd to NVMe.
Llama.cpp
- 32 t/s decode / 463 t/s prefill at UD-Q4KXL with a 640K serving pool (4x160K lanes, q8 KV)
- The 51B table shows 0% cross-token reuse, self-regulating to near-zero residency at zero decode cost — SSD random reads are the right architecture, not a compromise
- --moe-cache 5000 (PR #24528 branch): +30% decode; concurrency erases it — fan out shallow, serialize deep
- Warm prefix caching reprocesses only 24 tokens per turn (DeltaNet state included)
- Found and fixed a gridDim.y overflow crash (same shape as pending upstream PR #27941, 2-line fix)
- cgroup write-caging (MemoryMax=6G) keeps 32 t/s while pulling 90GB
vLLM
- Grafted unmerged mmap-PLE PR (#54129): table served from NVMe, W4A16, TP=2 — official recipe wants ≥110GB free RAM, this runs in 53GB
- Real-table decode 15.3 t/s vs 16.0 stubbed (4% NVMe cost)
- Selective experts-only UVA offload is mandatory: naive --cpu-offload-gb recopies everything each step (2.5 t/s); routing routedexperts weights gives 16 t/s
- MTP speculation hurts when PCIe is the bottleneck (13.9 vs 16.0)
More from Infra
- Scobleizer: Qwen Cloud is natively built around AI for better agent integration — Scobleizer · 2026-08-29
- NVL72 Achieves Up to 30x Better Throughput per MW than GB300 on AgentX Benchmark — nvidia · 2026-08-29
- a16z Partner: Only 2% of US Electricians Certified for DC Power — GregCook2011 · 2026-08-29
- The 'Boring' Network That Saves GPU Training Runs: OOB Management Explained — AccBalanced · 2026-08-29
- Running Generalist Robot Policies on STM32 and ESP32 Chips — yacineMTB · 2026-08-29
- Privacy architecture builds user trust to share sensitive health data with AI — bgmshana · 2026-08-29