Qwen3.8-Flash-Next on 2x3090 + DDR4: expert cache PR lifts decode from 17 to 25-29 t/s
Extension-Bid-639 · reddit · 2026-09-03
A Reddit user shares detailed benchmarks running Qwen3.8-Flash-Next on 2x RTX 3090 (PCIe 3.0) + dual Xeon + 188GB DDR4 with llama.cpp and unsloth UD-Q6KXL at full 261k context.
Results: decode jumped from 17 t/s to 25-29 t/s at short/mid context (17 at 131k depth); prefill (350 t/s on 26k prompts) unchanged.
How: llama.cpp PR #27861's GPU-resident LRU expert cache — instead of parking whole expert layers in VRAM, it caches recently used experts per layer. Consecutive tokens mostly reuse experts (80-85% hit rate on code). Key tricks: freeing VRAM for the cache and dropping ubatch from 2048 to 512 (compute buffers shrink 5GB/GPU), raising slots per layer from 80 to 135. Cost: slower prefill on long prompts.
What didn't help: thread count/poll/CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7 (only wins greedy or deep context), >2 cache uploads per step (saturates PCIe 3.0).
Reproducible: full commands provided — clone llama.cpp, merge PR #27861, launch with --moe-expert-cache 135; size cache to leftover VRAM (100MB per slot per GPU at Q6). Q4KXL testing is next.
More from Infra
- RTX 5090 writes nightly stock briefs with a numbers gate so the LLM can't invent figures — JakeChj · 2026-09-03
- Loop Transformer's Real Winner Is SRAM-Based Ultrafast Inference, Argues Bing Xu — bingxu_ · 2026-09-03
- CXMT reaches 10% global DRAM market share in Q2, Counterpoint Research says — zephyr_z9 · 2026-09-03
- Open-Source RL Framework Miles Debuts for Enterprise LLM and VLM Post-Training — AravSrinivas · 2026-09-03
- You Don't Run a Model, You Run Kernels: Why Inference Performance Hides in Fused Kernels — EAccelerate_42 · 2026-09-03
- KAIST's Declarative Attention lets LLMs skip most KV cache reads — kaist-ai · 2026-09-03