KLQ: Training-free LLM Quantization Beating SpinQuant
Federal-Setting-3014 · reddit · 2026-08-10
An independent researcher introduced KLQ, a training-free quantization method from a summer project. At W4A4KV4-bits, it outperforms all training-free rotation-based methods and gets close to the trained ReSpinQuant on Llama 3.2 1B.
Core Approach
While traditional methods use generic or computationally intensive learnable rotations to even out the uneven embedding space, KLQ takes a different path:
- Measure, Don't Flatten: It measures the unevenness using causal KL divergence, ranking the eigenbasis directions by importance.
- Waterfilling Algorithm: Treating each direction as an independent channel, it applies the provably optimal waterfilling algorithm to allocate the most bit-width to the most important directions.
Limitations & Costs
- Compute Intensive: Quantization requires one forward pass per direction per matrix per layer. On a single RTX 3090, probing Qwen 2.5 0.5B took 5 hours, and Llama 3.2 1B took 10 hours.
- No Real Kernels: It is currently a theoretical framework with a "fake" quantization demo lacking real inference kernels.
The author is seeking feedback and contributions from the community.
More from Research
- ICML Paper: The Real Challenge for Superintelligence Is Coexistence, Not Capability — xuanalogue · 2026-08-10
- Scholars Propose AI-Driven Overhaul for Academic Peer Review — paulnovosad · 2026-08-10
- Quanta Magazine Explains How 'Concept Cells' Abstract Information in the Brain — burny_tech · 2026-08-10
- SupraLabs Releases SupraElegans-500K: A C. elegans-Inspired Non-Transformer LLM — Dangerous_Try3619 · 2026-08-10
- Rumors: Chinese open-weights models advanced by extracting reasoning traces from Claude Code and Codex — jxmnop · 2026-08-10
- From GPT-2 to Kimi3: A Deep Dive into LLM Architecture Evolution — iamrobotbear · 2026-08-10