ExLlamaV3 MoE offloading on 9070XT doubles prefill to 2606 t/s for Qwen3.6 35B
willeyh · reddit · 2026-11-17
Reddit user willeyh reports substantial gains tuning MoE expert offloading with ExLlamaV3 on a 9070XT 16GB: running Qwen3.6 35B A3B Q4 with experts offloaded to DDR4, raising chunk size to 5120 lifted prefill from 1300 to over 2000 t/s (peaking at 2606 t/s @ 32K) while decode stayed flat. cputhreads 6 helped slightly; swizzle/gdnsub/bggdn/wide flags all regressed; cpumoepinnedarena caused core dumps. Benchmarks and tables are on GitHub (moe-offload-bench), compiled for ROCm.
More from Infra
- Engineer runs Gemma 3 270M entirely inside an office laser printer at 60s per token — osanseviero · 2026-11-17
- Older GPUs abandoned by new inference engines: are they still worth keeping? — pipecanon · 2026-11-17
- Blogger argues total compute controls create blind spots and accelerate adversaries — TinfoilTricorn · 2026-11-17
- How rollout staleness hurts on-policy distillation, and how to build a faster pipeline — burkov · 2026-11-17
- SK hynix's Nvidia revenue doubled to $24B, but US customer concentration is the real risk — tengyanAI · 2026-11-17
- macOS TCP Stack Breaks After Exactly 49.7 Days of Uptime Due to 32-bit Timestamp Overflow — DanielLockyer · 2026-11-17