MoE expert offloading on a 16GB RX 9070XT: three engines benchmarked, ExLlamaV3 leads
willeyh · reddit · 2026-11-13
The author spent two days benchmarking MoE expert offloading across llama.cpp, ExLlamaV3 and Strata on an RX 9070XT (RDNA4, 16GB VRAM) with 32GB DDR4, targeting usable 128K context with Q8 KV. Verdict: 16GB can't fit a decent Q4 quant plus real context, but hot-expert caches in Strata, ExLlamaV3 and recent llama.cpp make offloading viable on slow system RAM. Best results at 128K: Ornith 1.5 35B-A3B (EXL3 4.0bpw, DFlash2/ExLlamaV3) at 1168 tok/s prefill / 57.3 tok/s decode; Qwen3.6-35B-A3B same setup at 1246/71.6; Qwen3.8-Flash-Next Coder (IQ1M, Polystrata, --spec 2) at 1572/44. Surprise: 35B prefill tanks with context length (1900 t/s at 16K → 700 t/s at 128K). Full graphs, multi-bpw comparisons and a Go bench suite are open-sourced at moe-offload-bench.
More from Infra
- Microsoft outage caused by unresponsive servers during change validation — Kyrannio · 2026-11-13
- 'Compute won't escape overbuild': trader John Arnold on the coming capacity glut — McDonaghMatthew · 2026-11-13
- AWS tests four Bedrock settings that cut inference cost: cache TTL, Flex tier, effort, maxTokens — AWS ML Blog · 2026-11-13
- Cloudflare Workers rate limiting now supports delta-based usage in limit() calls — ritakozlov · 2026-11-13
- RL engineering debate: decoupled rollout vs colocated vLLM, staleness is not a rounding error — willcb · 2026-11-13
- WakeGPU: open-source desktop app to run remote Colab GPUs with auto idle shutdown — Far-Ad9512 · 2026-11-13