Verified engine map for Qwen3.8-27B on consumer GPUs: 379 tok/s on two 5070 Tis, RAM costs more than the card
FeydRowan · reddit · 2026-10-19
The author of capy-flash mapped every engine that runs Qwen3.8-27B and Qwen3.8-Flash-Next (125B MoE, 6B active) on consumer hardware, marking verified runs vs. author-claimed numbers.
Architecture takeaways
- Multi-GPU: tensor parallelism is the only way to add bandwidth across cards for one request; pipeline splits add memory, not speed (+0–16%).
- MoE expert offloading: VRAM expert cache with CPU-computed misses dominates; the bottleneck is per-step expert misses. Mac setups stream experts from SSD.
- Quantization: NVFP4 is higher quality; 2–3-bit IQ/GSQ is faster with 2× the KL (0.144 vs 0.070 in Strata's own tests); EXL3 wins quality-per-bit.
- Speculative decoding: MTP gives 1.3–1.7×; DFlash2/DSpark drafters reach 2–3× — the source of most "400+ tok/s" headlines. A DFlash2 drafter for Flash-Next shipped but is unverified on consumer hardware.
- Hand-written single-model kernels beat general engines 2–6× at batch 1; vLLM/SGLang win with many concurrent users.
Representative numbers
- 27B decode: capy-flash on 2× 5070 Ti (TP2 + DFlash2) verified at 379 tok/s; llama.cpp on one 5090 verified at 205; tqLLM on a 3090 verified at 273.
- 27B prefill: NInfer claims 12.8k tok/s on a 5090; capy-flash verified 9.2k on 2× 5070 Ti (non-P2P boards).
- Rule of thumb: 1.8 TB/s bandwidth plus a good drafter caps batch-1 speed at 300–400 tok/s.
Cost note: at 2026 DDR5 prices, the RAM needed for Flash-Next costs more than the GPU — the author redid the cost math to include memory.
Related event: Developer Benchmarks Qwen3.8 Models on Consumer GPUs(2 posts)→
More from Infra
- Self-hosted DeepSeek inference beats every vendor on price/perf, boosting the open-weight enterprise case — bendee983 · 2026-10-19
- DeepSeek V4.1 Flash at 2.61 bpw hits 26 tok/s on dual RTX 3090s with dsx — whiteh4cker · 2026-10-19
- Wan 2.2 I2V on a 12GB GPU uses only 8-9GB VRAM—system RAM is the real bottleneck — noise_to_motion · 2026-10-19
- New video explains how to train models too big for one GPU: ZeRO and FSDP demystified — jbhuang0604 · 2026-10-19
- Running Qwen3-Flash-Next locally for coding: 90GB quantized weights, prefill pain at 7-8M tokens — intermundia · 2026-10-19
- Verified local LLM engine benchmarks: Qwen3.8-27B up to 540 tok/s on a single RTX 5090 — FeydRowan · 2026-10-19