exllamav3 CPU-offload beats llama.cpp 3.2x prefill, 2x decode on Qwen

Lowkey_LokiSN · reddit · 2026-09-08

On a 2×RTX 3080 20GB + 128GB DDR4 + Xeon 6148 setup, llama.cpp running Unsloth's Q4KXL quant of Qwen-3.8-Flash-Next managed 270tps prefill and 13tps decode (degrading with context).
exllamav3's 4.05 EXL3 quant hit 25tps average decode (peaks of 32tps) stable through 160k context, with 870tps prefill — 3.2x faster prefill, 2x faster decode, and better quant quality.
The win doesn't generalize: GLM 5.3 Flash's EXL3 ran 2x slower than llama.cpp, with AVX2 as the bottleneck; results depend on model architecture and hardware.
Practical notes: exllamav3 + TabbyAPI is trickier to configure, and decode speeds warm up over a few thousand tokens (12tps rising to 25-30) as hot/cold experts calibrate.

More from Infra