Qwen3.8-Next-Flash hits ~240 t/s decode on a single RTX 6000 Pro
AdventurousSwim1312 · reddit · 2026-08-31
Building on jpezzulli's patched sglang (178 t/s), a Reddit user let an AI iterate for days and pushed Qwen3.8-Flash-Next-NVFP4 decode to nearly 240 t/s on a single RTX 6000 Pro MaxQ (300W), vs a theoretical nvfp4 bandwidth limit around 280 t/s.
Tricks used:
- Further quantizing lm head and several layers from bf16 to fp8 to cut bandwidth
- Kernel tuning of selected layers for GPU efficiency
- MTP config tuning
Next up: full nvfp4 layer quantization, layer-fusion kernels, MTP post-training and an eagle3 head. Repro resources (patch repo, base repo, model checkpoint) are all open. Test rig: Ryzen 9 3950X, 128GB RAM (ngram table), 1x RTX 6000 Pro MaxQ.
More from Infra
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16