Custom IQ3 Requant for DeepSeek-V4-Flash: 1.4x Decode Speed on Mixed-GPU Rigs
HockeyDadNinja · reddit · 2026-08-02
A developer shared a custom quantization recipe (GGUF) for DeepSeek-V4-Flash-0731, optimized for mixed multi-GPU setups where experts spill into system RAM.
Quantization Strategy
Only the 129 routed expert tensors were requantized to IQ3XXS (down projections at IQ3S), while other tensors retained their original high precision from the MXFP4 source. The final size is 111.37 GiB.
Quality & Performance
- Quality: Compared to unsloth's UD-IQ3S, this recipe achieves better mean KLD (0.2386 vs 0.2936) and top-1 agreement, though it is slightly larger and has a worse max KLD.
- Speed: On a 5-GPU mixed rig (96GB VRAM with CPU spill), it delivers 13.91 t/s, offering a 1.4x decode speedup over the MXFP4 source.
The author notes that smaller Q2 quantizations are faster if purely chasing tokens per second. They also warn of known SWA and rollback stall issues with DeepSeek-V4-Flash in mainline llama.cpp, requiring patched builds to avoid stalls.
Related event: DeepSeek-V4-Flash Local Deployment Benchmarks: Performance Across Hardware(21 posts)→
More from Infra
- Running 2.78T Parameter Kimi K3 on a Single CPU with 8GB RAM — Saboo_Shubham_ · 2026-08-03
- AMD Enters Open-Source LLM Arena with Instella-MoE-16B — airesearch12 · 2026-08-03
- US States Move to Repeal Data Center Tax Breaks, Raising AI Infrastructure Costs — pstAsiatech · 2026-08-03
- Handling Offline AI Jobs: Developers Share Best Engineering Practices — cmm324 · 2026-08-03
- A 10-Week Roadmap for LLM Inference Serving and Optimization — _jaydeepkarale · 2026-08-03
- App Developers Should Ship Their Own On-Device Models — abacaj · 2026-08-03