Qwen3.8-Flash-Next on 2x3090 + DDR4: expert cache PR lifts decode from 17 to 25-29 t/s
Extension-Bid-639 · reddit · 2026-09-03
A Reddit user shares detailed benchmarks running Qwen3.8-Flash-Next on 2x RTX 3090 (PCIe 3.0) + dual Xeon + 188GB DDR4 with llama.cpp and unsloth UD-Q6KXL at full 261k context.
Results: decode jumped from 17 t/s to 25-29 t/s at short/mid context (17 at 131k depth); prefill (350 t/s on 26k prompts) unchanged.
How: llama.cpp PR #27861's GPU-resident LRU expert cache — instead of parking whole expert layers in VRAM, it caches recently used experts per layer. Consecutive tokens mostly reuse experts (80-85% hit rate on code). Key tricks: freeing VRAM for the cache and dropping ubatch from 2048 to 512 (compute buffers shrink 5GB/GPU), raising slots per layer from 80 to 135. Cost: slower prefill on long prompts.
What didn't help: thread count/poll/CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7 (only wins greedy or deep context), >2 cache uploads per step (saturates PCIe 3.0).
Reproducible: full commands provided — clone llama.cpp, merge PR #27861, launch with --moe-expert-cache 135; size cache to leftover VRAM (100MB per slot per GPU at Q6). Q4KXL testing is next.
Related event: Tuning Qwen3.8-Flash-Next on 2x3090: From 17 to 41 t/s Decoding(2 posts)→
More from Infra
- Developer Moves AI Agents to a Dedicated Linux Workstation Using Incus System Containers — kevinkern · 2026-10-21
- Nvidia guides 70% growth citing unmet AI demand, while media panics over a $500M Korean neocloud IPO — firstadopter · 2026-10-21
- Pichai: everyone is massively short on AI compute — 'everyone wishes you had two times what you had planned' — firstadopter · 2026-10-21
- Stanford CS336 Lecture 6: fused Triton kernels cut softmax memory traffic ~4x — stanfordnlp · 2026-10-21
- General Compute partners with Positron AI: Atlas memory-centric hardware ships Q1 2027 — aftahi_ai · 2026-10-21
- AI bond issuance hits $419B in 2026, up 90% from all of 2025 — SumitGup · 2026-10-21