Qwen3 at 2.7 bits/weight with Leech lattice quantization beats IQ2_S on quality, loses on speed
Fuzzy-Layer9967 · reddit · 2026-11-15
An independent Rust implementation of LLVQ (Leech lattice quantization, arXiv:2603.11021) with custom CUDA kernels quantizes Qwen3 4B/8B/14B to 2.70–2.73 bits per parameter, with weights on the Hub.
How it works: 24 weights are jointly snapped to the nearest point of the 24-dim Leech lattice, encoding each block as a 48-bit code (2 bits/weight). The decoder reads codes directly during matvec without building a dense matrix; a whole token runs in one CUDA launch, kernels compiled at runtime via NVRTC.
Same-bits comparison (same calibration set, same L40S, same day vs. IQ2S): MMLU +9.1/+4.1/+2.9 at 4B/8B/14B; GSM8K +14.3/+3.0/tied; WikiText-2 perplexity 12.58/9.71/8.44 vs 16.78/10.75/9.47. The gap shrinks as models grow.
Same-memory, one size up: 2.7-bit 8B beats 4-bit AWQ 4B by 1.44 MMLU points; 14B beats AWQ 8B by 1.87 with 1GB less memory, GSM8K ties both times.
Costs: vs FP16, 4B loses 6.8 MMLU / 9.6 GSM8K points; notably slower than llama.cpp (34–64 tok/s behind IQ2S), and L40S is the only card with a real gain. The author invites scrutiny and discloses the Unsloth 4B file loops under greedy decoding.
More from Infra
- Upgrading from GTX 1070: Is RTX 3060 12GB Enough for Local Image Gen and LLMs? — Guilty_Elk4385 · 2026-11-15
- Epoch AI: post-training enhancements yield gains equal to 5-20x compute at under 1% the cost — CFGeek · 2026-11-15
- What 'Compute Multipliers' Mean: The Under-discussed Algorithmic Efficiency Metric in AI — CFGeek · 2026-11-15
- CoreWeave says GPU pricing jumped 30-50% QoQ, above original contract rates — Beth_Kindig · 2026-11-15
- Americans: no data centers in our backyard! SpaceX: no problem — draecomino · 2026-11-15
- The Starship math: 25,000 flights to deploy 250GW of space compute — JOBhakdi · 2026-11-15