DeepSeek V4 Quantization: Fixing Conversion Pitfalls and 8x RTX 5090 Benchmarks
gladkos · reddit · 2026-08-12
Developers quantized DeepSeek V4 0731 on 8x RTX 5090s and benchmarked against popular quants. They identified two conversion issues: omitting --no-lazy causes NaN weights, and the default FP8 to Q80 downconversion introduces significant deviation.
By replacing these tensors with BF16, they created a bit-exact baseline. They then built 13 quants using imatrix, noting that cross-comparisons are difficult due to non-standardized naming and GPU-specific fast paths.
Ultimately, they tested 38 files on a single machine. In the 85-135 GB range, their quantization curve shows less drift than community equivalents. For 128 GB hardware, they recommend their 104 GB AD-IQ2M quant.
More from Infra
- Hyperscalers Still Rely on 2017's V100s: AI Compute Lifespan Reaches 9 Years — BenBajarin · 2026-08-12
- TensorScale Unveils Fastest Video Inference, Claims 10x Speedup for MiniMax H3 — Scobleizer · 2026-08-12
- vLLM and NVIDIA Co-host Meetup on Scaling LLM Inference Efficiency — vllm_project · 2026-08-12
- Napkin Math for AI: A Practical Trick to Diagnose Inference Bottlenecks — HamelHusain · 2026-08-12
- AI Infrastructure Boom Drives Near Triple-Digit Revenue Growth — sudoraohacker · 2026-08-12
- CMP170HX Mining Cards Tested: 4x64GB for LLM Inference, Performance and Value Analysis — m94301 · 2026-08-12