SLQ quantizes LLMs to 3.3 bits per parameter and still speeds up inference
TheZachMueller · x · 2026-07-24
Statistically-lossless quantization reaches 3.3 bits per parameter
This paper proposes SLQ (Statistically-Lossless Quantization) for large language models and argues that quantization does not have to force a trade-off between fidelity and speed.
- Task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and can be achieved at aggressive bit widths.
- Distribution-lossless compression is defined more strictly: the quantized model’s next-token distribution should be practically indistinguishable from the original.
- The authors introduce Expected Acceptance Rate (EAR) as an interpretable fidelity metric; for example, EAR ≥ 0.99 implies 99% token agreement under optimal coupling.
- They prove a variance law showing symmetric quantization inflates noise variance by γ² relative to asymmetric quantization.
- With SLQ, they report task-lossless compression below 4 bpp (as low as 3.3 bits/parameter, model-dependent), distribution-lossless compression at 5–6 bpp, and 1.7–3.6× inference speedups versus FP16 using optimized kernels.
The code is published at github.com/IST-DASLab/SLQ.
Related event: SLQ Paper Achieves Near-Lossless LLM Quantization with Speedup(2 posts)→
More from Infra
- A100 rental prices still rising in 2026 per a16z data, Jevons paradox hits compute — VraserX · 2026-10-27
- Anthropic S-1 reveals $800B+ in off-books future compute commitments — GaryMarcus · 2026-10-27
- ~42% of Anthropic's ~11GW contracted compute is NVIDIA GPUs — ns123abc · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- RTX 5080 price jumps from $1658 to $2318 in ten days on Amazon — Frizzy-MacDrizzle · 2026-10-27
- ComfyUI on 12GB VRAM: measured peaks show offloading turns 24s into 11 minutes — Impossible-Mail-4619 · 2026-10-27