Kicking expert cache off GPU during prefill yields 2.2-2.5x speedup on 2x3090 MoE inference
Extension-Bid-639 · reddit · 2026-09-10
Part 4 of a Reddit series on running Qwen3.8-Flash-Next locally via llama.cpp (UD-Q4KXL quant, MTP head) on 2x3090 + dual Broadwell Xeon with experts pinned in host RAM. This installment tackles the long-standing weak spot: prefill, which took 80+ s to first token on an 8k prompt and 24 minutes at 119k context.
Key insight: the expert cache only serves decode batches (≤8 tokens), yet holds VRAM during prefill. The fix: when a prompt arrives, the server drops cache slots, decode compute buffers and CUDA pools, grabs ub-2048 compute buffers, runs the whole prompt, then restores everything before the first generated token. This works because expert weights stream over PCIe once per micro-batch regardless of size — at ub 512 an 8k prompt drags experts over PCIe 16 times vs 4 at ub 2048.
Measured (6x32GB DDR4, fresh server per arm): 8k prefill 99.9 → 223.7 t/s (2.24x), TTFT 82s → 37s; 37k context 2.41x; 119k context 2.54x (TTFT 1461s → 575s). Cost: 2.8s fixed release/restore overhead per prompt; decode essentially unchanged (±2%), MTP acceptance stable at 0.79-0.83, quality screens within ±3.6%. Enabled by two env vars; the swap is transactional — a failed restore errors out rather than silently limping.
More from Infra
- Meta signs deal for tens of millions of AWS Graviton cores to power agentic AI — bookwormengr · 2026-09-10
- Edge0 open-sources streaming MoE framework with 35B and 8B on-device models — EAccelerate_42 · 2026-09-10
- iPhone 18 Pro's variable aperture: Apple returns to physics after hitting compute wall — r0ck3t23 · 2026-09-10
- Flux 2 Klein image model now runs fully in the browser via WebGPU — radamar · 2026-09-10
- AI server rail maker King Slide's founder is now Taiwan's richest man at $17.8B, passing Amodei and Altman — rwang07 · 2026-09-10
- SF Compute says resale of idle capacity saves customers 25%, cutting $4.5/hr to $3.375/hr — mattshumer_ · 2026-09-10