Qwen3.8-Flash-Next on a single R9700: 863 t/s prefill and 35 t/s decode at 230k context
Designer_Elephant227 · reddit · 2026-10-02
A Redditor shared full details of running Qwen3.8-Flash-Next locally on a single AMD R9700 via the exllamav3 rocm fork, asking whether the setup is optimal.
- Model: turboderp's exl3 5.05 bpw quant (6-bit head/vision, 5-bit mtp)
- Expert offload: 112 of 512 experts per layer on GPU, 400 on CPU (16 threads); a 102GB bf16 ngram table streamed from NVMe
- 262k context, q8 KV cache, chunk size 4096, MTP drafting enabled (53-54% acceptance)
- Measured at 230k context prose: 863 t/s prefill (101k new tokens, rest from prefix cache), 34.7 t/s decode
- Required patching one file for gfx12
More from Infra
- tinygrad lists $89,000 tinybox with 768GB VRAM: four refurbished AMD MI300X GPUs — viksit · 2026-11-19
- RustingStrata vs Strata on RTX 3060: Rust MoE engine edges out 28.3 vs 26.1 tok/s — Serious_Scarcity7818 · 2026-11-19
- Investor: market hates uncertainty more than money after Nvidia earnings — MartinGTobias · 2026-11-19
- SGLang ships Day-0 serving support for new model DaisyRose-27B — ying11231 · 2026-11-19
- Linux LTS patch flood: 3318 patches for 6.6 alone, 'no way humans are reviewing this' — pkqzy888 · 2026-11-19
- DaisyRose-27B's in-house sandbox: ~80% cheaper than cloud providers, 99.95% success rate — BanghuaZ · 2026-11-19