KQuant: open-source MLX quants cut Qwen KL divergence up to 3.7x vs public 4-bit
m4rkGS · reddit · 2026-11-12
Alish, an ML engineer at Nace (the team behind the open-source Kapitsa Mac runner), released KQuant, an open-source quantization method producing MLX packs for the Qwen family that stay far closer to full precision than community 4-bit quants.
Results (KL divergence from the full model, lower is better)
- Qwen3.8-Flash-Next: 0.067 vs 0.249 public 4-bit — 3.7x better and 5GB smaller (105.2GB).
- Qwen3.5-9B: 0.030 vs 0.081 (2.6x); 4B 1.45x, 27B 1.5x, 35B-A3B 1.6x better. Q5 variants drop further (9B Q5: 0.010).
- Speed: on M5 Pro with Kapitsa's KFlash drafter, Q4 writes 6–15% faster than public 4-bit packs.
Method: real-text-guided search for per-group scales/offsets, with extra bits allocated to sensitive layers; search runs on cloud GPU clusters before release. Output is standard MLX files — no custom kernels or loaders. GGUF packs are coming, and the author is running an AMA while recruiting early GGUF testers.
More from Infra
- Nebius Q3: $12B in customer commitments, revenue up 68% to $981M as GPU prices climb 22% — demian_ai · 2026-11-12
- 1T-Parameter MoE trained on EU public supercomputer with PyTorch, per-GPU throughput up 14x — PyTorch · 2026-11-12
- llama.cpp crosses 134k GitHub stars as the default local LLM inference stack — ggml-org · 2026-11-12
- TensorFold: a Zig LLM inference engine spanning Metal, CUDA and Vulkan — ashhart · 2026-11-12
- TinyTapeout Explains How Open-Source Chips Are Factory-Tested Before Shipping — Obijuan_cube · 2026-11-12
- Evolving Infra: agent-adaptive LLM serving cuts latency up to 4.1x vs NVIDIA Dynamo — mit-han-lab · 2026-11-12