New SLQ paper claims near-lossless LLM quantization with 1.7×–3.6× speedups

pmttyji · reddit · 2026-07-25

This paper argues for a middle ground between lossy and fully lossless LLM quantization: statistically-lossless compression.

It defines task-lossless compression as preserving zero-shot accuracy within natural sampling variance, and shows this can hold even at aggressive bit widths.
It introduces a stricter notion of distribution-lossless compression, where the quantized model’s next-token distribution is practically indistinguishable from the original.
The paper proposes Expected Acceptance Rate (EAR) as an interpretable fidelity metric, where EAR ≥ 0.99 means 99% token agreement under optimal coupling.
It also proves a gamma-squared variance law: symmetric quantization inflates noise variance by γ² relative to asymmetric quantization, implying asymmetry is necessary for distribution-lossless fidelity.
Using SLQ, a layer-wise non-uniform asymmetric method, the authors report task-lossless compression below 4 bits/parameter in some models, distribution-lossless compression at roughly 5–6 bits/parameter, and 1.7×–3.6× inference speedups over FP16 with optimized kernels.

More from Infra