exllamav3 CPU-offload beats llama.cpp 3.2x prefill, 2x decode on Qwen
Lowkey_LokiSN · reddit · 2026-09-08
- On a 2×RTX 3080 20GB + 128GB DDR4 + Xeon 6148 setup, llama.cpp running Unsloth's Q4KXL quant of Qwen-3.8-Flash-Next managed 270tps prefill and 13tps decode (degrading with context).
- exllamav3's 4.05 EXL3 quant hit 25tps average decode (peaks of 32tps) stable through 160k context, with 870tps prefill — 3.2x faster prefill, 2x faster decode, and better quant quality.
- The win doesn't generalize: GLM 5.3 Flash's EXL3 ran 2x slower than llama.cpp, with AVX2 as the bottleneck; results depend on model architecture and hardware.
- Practical notes: exllamav3 + TabbyAPI is trickier to configure, and decode speeds warm up over a few thousand tokens (12tps rising to 25-30) as hot/cold experts calibrate.
More from Infra
- SpaceX strikes $40B deal with Apollo Global to lease AI chips and data center hardware — Polymarket · 2026-10-26
- India Needs at Least 5GW of Compute by 2028 to Stay Relevant, User Argues — EAccelerate_42 · 2026-10-26
- SpaceX AI revenue up 830%, Claude designs 318K antivenom proteins in 64 hours — PeterDiamandis · 2026-10-26
- Meta plans up to $95 billion stock sale to fund AI buildout — The Decoder · 2026-10-26
- Pandora M1: first prototype of a physical device for distributed GPU monitoring — tarantulae · 2026-10-26
- INT21's AI agent swarm rewrote Drama 3's inference runtime in Rust within one hour, beating SGLang-Omni — bingxu_ · 2026-10-26