Dual RTX 5060 Ti Only Gets 10 t/s on Qwen3.8-Flash-Next, Seeking Config Advice
MkGod · reddit · 2026-09-19
A Reddit user is trying to get usable speeds running Qwen3.8-Flash-Next-UD-Q3KXL via Unsloth Studio (llama-server) on dual RTX 5060 Ti 16GB (32GB total VRAM) with 96GB DDR4.
Tried so far:
- CPU-offloaded experts via --cpu-moe -ngl 99 -b 1024 -ub 512 -fa on
- 65k context, q40 KV cache
- MTP speculative decoding (87% draft acceptance)
Decode is only 9.5–10 t/s with 12–38 t/s prefill. Others squeeze 25–40 t/s from dual 3090s using the new expert-cache PRs (#27861/#28223) and pinned host memory, but he's unsure of the best path on 32GB VRAM + DDR4: drop MTP entirely, bypass mmap with --load-mode none, build a custom llama.cpp with LRU expert cache—or is he fundamentally bandwidth-capped?
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04