4x R9700 EXL3 serving: 1,094 tok/s Qwen decode, all optimization tricks open-sourced
mikoskinen · reddit · 2026-11-13
A detailed writeup of optimizing EXL3 serving of DeepSeek 4.1 Flash (2.25bpw) and Qwen Flash-Next (4.05bpw) on 4x AMD R9700 + EPYC Rome, targeting Claude Code workloads with byte-identical outputs.
Results: at 16 concurrent users, DS4.1F decodes at 259 tok/s and QFN at 1,094 tok/s; QFN prefill hits 9,540 tok/s at 16k (TTFT 1.7s), DS4.1F reaches 6,140 tok/s at 128k.
Software optimizations
- DS4.1F's upper 20 of 40 layers only read the prompt tail: 136k cold prefill cut from 50.5s to 33.3s (-34%)
- Interleaved push in the 4-card all-reduce: 42MB all-reduce from 3.4ms to 2.5ms (-27%)
- QFN splits hyper-connection stages across cards: +5.7%
Hardware optimizations
- Same-size DIMMs in one interleave set: CPU read bandwidth 35.9 → 104 GB/s
- One GPU per IO-die quadrant: host-to-GPU copies 61 → 105 GB/s
- Disabling deep C-states + performance governor removed a random 16% slowdown
All testing, profiling and coding were done by Claude Opus. Full results and tips are open-sourced at github.com/mikoskinen/r9700-exact-serving.
More from Infra
- One DGX Spark runs ripple-65b preview at 10.8 tok/s with coherent long reasoning — DZhang50 · 2026-11-13
- Qwen3.8-Flash-Next: 180B total / 6B active, runs on consumer GPUs via Strata offloading — drdanielbender · 2026-11-13
- Plotly's DataAnalyticsBench: three models that fit in 64GB score a B on 43 analytics tasks — DZhang50 · 2026-11-13
- Hugging Face CEO calls Jensen Huang's Nvidia resignation 'sad', vows to keep fighting for open source — nordicinst · 2026-11-13
- Boole-32B-A2.5B-preview: 9GB 1.8-bit MoE hits 565 token/s on a single RTX 5090 — d77chong · 2026-11-13
- llama.cpp bumps to version 1.0.0, ending its long 0.x era — jacek2023 · 2026-11-13