New SLQ paper claims near-lossless LLM quantization with 1.7×–3.6× speedups
pmttyji · reddit · 2026-07-25
This paper argues for a middle ground between lossy and fully lossless LLM quantization: statistically-lossless compression.
- It defines task-lossless compression as preserving zero-shot accuracy within natural sampling variance, and shows this can hold even at aggressive bit widths.
- It introduces a stricter notion of distribution-lossless compression, where the quantized model’s next-token distribution is practically indistinguishable from the original.
- The paper proposes Expected Acceptance Rate (EAR) as an interpretable fidelity metric, where EAR ≥ 0.99 means 99% token agreement under optimal coupling.
- It also proves a gamma-squared variance law: symmetric quantization inflates noise variance by γ² relative to asymmetric quantization, implying asymmetry is necessary for distribution-lossless fidelity.
- Using SLQ, a layer-wise non-uniform asymmetric method, the authors report task-lossless compression below 4 bits/parameter in some models, distribution-lossless compression at roughly 5–6 bits/parameter, and 1.7×–3.6× inference speedups over FP16 with optimized kernels.
Related event: SLQ Paper Achieves Near-Lossless LLM Quantization with Speedup(2 posts)→
More from Infra
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Lam Research: weekly token usage up ~250x in 18 months — Beth_Kindig · 2026-10-27
- AWS shows how to build enterprise agentic retrieval on Bedrock with CDK — AWS ML Blog · 2026-10-27
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27