New SLQ paper claims near-lossless LLM quantization with 1.7×–3.6× speedups
pmttyji · reddit · 2026-07-25
This paper argues for a middle ground between lossy and fully lossless LLM quantization: statistically-lossless compression.
- It defines task-lossless compression as preserving zero-shot accuracy within natural sampling variance, and shows this can hold even at aggressive bit widths.
- It introduces a stricter notion of distribution-lossless compression, where the quantized model’s next-token distribution is practically indistinguishable from the original.
- The paper proposes Expected Acceptance Rate (EAR) as an interpretable fidelity metric, where EAR ≥ 0.99 means 99% token agreement under optimal coupling.
- It also proves a gamma-squared variance law: symmetric quantization inflates noise variance by γ² relative to asymmetric quantization, implying asymmetry is necessary for distribution-lossless fidelity.
- Using SLQ, a layer-wise non-uniform asymmetric method, the authors report task-lossless compression below 4 bits/parameter in some models, distribution-lossless compression at roughly 5–6 bits/parameter, and 1.7×–3.6× inference speedups over FP16 with optimized kernels.
Related event: SLQ Paper Achieves Near-Lossless LLM Quantization with Speedup(2 posts)→
More from Infra
- GPT-5.6 Agents Collaborate for 40 Hours to Boost Kimi K3-like Model Inference to 406 tok/s — nicodotdev · 2026-07-25
- llama.cpp runs faster on E-cores than P-cores in a GPU-offloaded MoE test — dir3ctly · 2026-07-25
- Artificial Analysis says Opus 5 is not much cheaper to use than Fable — WonderFactory · 2026-07-25
- AlphaFold is being used to redesign gene-editing proteins for safer edits — Steap-Edit · 2026-07-25
- NVIDIA Dynamo adds Squeeze Evolve, claiming up to 3x lower cost and 10x throughput — james_y_zou · 2026-07-25
- Developers still struggle to get SM 12.0 GPUs into CI and GitHub Actions — SkyLi0n · 2026-07-25