Custom IQ3 Requant for DeepSeek-V4-Flash: 1.4x Decode Speed on Mixed-GPU Rigs
HockeyDadNinja · reddit · 2026-08-02
A developer shared a custom quantization recipe (GGUF) for DeepSeek-V4-Flash-0731, optimized for mixed multi-GPU setups where experts spill into system RAM.
Quantization Strategy
Only the 129 routed expert tensors were requantized to IQ3XXS (down projections at IQ3S), while other tensors retained their original high precision from the MXFP4 source. The final size is 111.37 GiB.
Quality & Performance
- Quality: Compared to unsloth's UD-IQ3S, this recipe achieves better mean KLD (0.2386 vs 0.2936) and top-1 agreement, though it is slightly larger and has a worse max KLD.
- Speed: On a 5-GPU mixed rig (96GB VRAM with CPU spill), it delivers 13.91 t/s, offering a 1.4x decode speedup over the MXFP4 source.
The author notes that smaller Q2 quantizations are faster if purely chasing tokens per second. They also warn of known SWA and rollback stall issues with DeepSeek-V4-Flash in mainline llama.cpp, requiring patched builds to avoid stalls.
Related event: DeepSeek V4 Flash Sparks a Wave of Local Deployment Tests(26 posts)→
More from Infra
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- GYRO quant of Qwen3.8-Flash-Next tested: near Q3 quality at much smaller size — PlasticRevenue4601 · 2026-10-16
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16