New SLQ paper claims near-lossless LLM quantization with 1.7×–3.6× speedups
pmttyji · reddit · 2026-07-25
This paper argues for a middle ground between lossy and fully lossless LLM quantization: statistically-lossless compression.
- It defines task-lossless compression as preserving zero-shot accuracy within natural sampling variance, and shows this can hold even at aggressive bit widths.
- It introduces a stricter notion of distribution-lossless compression, where the quantized model’s next-token distribution is practically indistinguishable from the original.
- The paper proposes Expected Acceptance Rate (EAR) as an interpretable fidelity metric, where EAR ≥ 0.99 means 99% token agreement under optimal coupling.
- It also proves a gamma-squared variance law: symmetric quantization inflates noise variance by γ² relative to asymmetric quantization, implying asymmetry is necessary for distribution-lossless fidelity.
- Using SLQ, a layer-wise non-uniform asymmetric method, the authors report task-lossless compression below 4 bits/parameter in some models, distribution-lossless compression at roughly 5–6 bits/parameter, and 1.7×–3.6× inference speedups over FP16 with optimized kernels.
Related event: SLQ Paper Achieves Near-Lossless LLM Quantization with Speedup(2 posts)→
More from Infra
- EdgeRock open-sourced: 72 MiB KV cache at 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- RTX 5080 price jumps from $1658 to $2318 in ten days on Amazon — Frizzy-MacDrizzle · 2026-10-27
- ComfyUI on 12GB VRAM: measured peaks show offloading turns 24s into 11 minutes — Impossible-Mail-4619 · 2026-10-27
- Neon rebuilds its docs search entirely on its own stack: Lakebase Search, Functions, AI Gateway — matei_zaharia · 2026-10-27
- PyTorch details 2.12x faster on-device streaming ASR on Arm SME2 via ExecuTorch — PyTorch · 2026-10-27