Verified local LLM engine benchmarks: Qwen3.8-27B up to 540 tok/s on a single RTX 5090

FeydRowan · reddit · 2026-10-19

The author, developer of the capy-flash engine, compiled a list of inference engines that run Qwen3.8-27B and Flash-Next (125B MoE, 6B active) on consumer hardware, distinguishing verified runs from self-reported numbers (test box: 2× RTX 5070 Ti + 96 GB DDR5).

Key takeaways

Tensor parallelism adds bandwidth across cards for one request; layer-splitting only adds memory
For Flash-Next, expert offloading matters: hot experts in VRAM, misses computed on CPU — misses per step is the key metric
Quant trade-off: NVFP4 is better quality but heavier; 2–3-bit IQ GGUF is faster at roughly double the KL (0.144 vs 0.070)
Drafters drive headline numbers: MTP gives 1.3–1.7×, DFlash2/DSpark give 2–3× on 27B, explaining most 400+ tok/s claims
Single-model kernels beat general engines by 2–6× at batch 1; vLLM/SGLang win at high concurrency

Notable numbers (27B): MegaCapybara 313–540 tok/s on one 5090 (unverified); capy-flash 379 tok/s verified on 2× 5070 Ti; tqLLM 273 tok/s on a 3090 (verified).

Cost analysis (EU street prices): 2× R9700 is the best value for Flash-Next at €4.4k / 272 tok/s (62 tok/s per €1k), while 2× 5090 is the worst at €19k / 142 tok/s (7). The author also replayed a real 53-turn coding session growing to 81k tokens to compare prefill and total latency across engines.

Original post →

Verified local LLM engine benchmarks: Qwen3.8-27B up to 540 tok/s on a single RTX 5090

More from coding & agent