GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan, Michael Helcig, Eldar Kurtic, Maximilian Kleinegger, Dan Alistarh
cs.CL, cs.LG
2026-04-21
GSQ learns scalar grid assignments with Gumbel-Softmax. Llama-3.1-70B at 2.13 bpp scores 75.57, +4.14 vs EfficientQAT and 1.68 behind QTIP; Qwen3-8B Q2_K GGUF rises 50.03 to 56.28.
Weight-only quantization for local serving is stuck between two waves. Scalar methods such as GPTQ and AWQ are simple and kernel-friendly, then accuracy falls off a cliff below 3–4 bits. Vector and trellis methods such as QTIP, AQLM, and GPTVQ push the 2–3-bit frontier, but the formats are hard to implement, decode barely faster than BF16, and do not scale to trillion-parameter MoE. The open question is whether that gap is the scalar grid itself, or the way the grid has been optimized.
A team at ISTA, ETH Zürich, and Red Hat AI argues for the second reading. GSQ (Gumbel-Softmax Quantization) treats per-weight grid assignment as a differentiable discrete problem and learns it jointly with per-group scales. The deployed object is still a symmetric, group-wise scalar grid, so existing scalar kernels can run it.
The objective is output reconstruction error on calibration inputs, subject to weights landing on a discrete grid. Each coordinate gets trainable logits; Gumbel-Softmax turns them into a soft mixture over grid points, then temperature annealing collapses the mixture onto one point. Ternary weights factor into a mask and a sign, which halves the logit count. A 2-bit uniform grid has four points, so four logits per weight plus a shared scale. Above 2 bits the grid grows exponentially, so GSQ switches to a local shift: only five moves {−2,−1,0,+1,+2} around the GPTQ initialization, cutting logits per weight from 8 to 5 at 3 bits. An ablation of the full 8-way relaxation finds 99.999996% of assignments already sit in that neighborhood.
The optimizer is Lion, because a saturated Gumbel-Softmax kills gradients and stalls AdamW. Inside a Transformer block, GSQ does not jointly train every linear under one block loss. It stages the work: Q and K first under linear reconstruction, then V and O under attention-output reconstruction, then the MLP under the full block loss. Finished blocks freeze, later blocks see the already-quantized prefix, so error accumulation is visible. 2-bit Llama runs one extra end-to-end pass that updates only group scales. Initialization is GPTQ, or an existing GGUF K-Quant checkpoint that is written back in the same format. Non-uniform bit allocation is handed to RCO, a companion searcher from the same lab.
Llama-3.1 zero-shot averages on ARC-C/E, HellaSwag, PIQA, and WinoGrande:
| Model | Method | bit/param | Avg |
| 8B | EfficientQAT / GSQ / QTIP | 2.25 / 2.13 / 2.00 | 63.79 / 68.55 / 69.88 |
| 70B | EfficientQAT / GSQ / QTIP | 2.25 / 2.13 / 2.00 | 71.43 / 75.57 / 77.25 |
| 70B | GSQ / QTIP | 3.13 / 3.00 | 77.99 / 78.17 |
At 2 bits GSQ beats the best scalar baseline, EfficientQAT, by 4.76 (8B) and 4.14 (70B), trails QTIP by 1.33 and 1.68, and trails PV-Tuning by 0.70. Those baselines use asymmetric quantization with zero-points; GSQ does not. Ternary 8B at 1.71 bpp scores 63.44, already past 2-bit GPTQ/QuIP and even with 2-bit EfficientQAT. On Llama-3.1-70B with vLLM plus Humming kernels, uniform 2-bit is 6.20× BF16 throughput on L40s.
Kimi-K2.5 quantizes only non-shared experts to 2.13 bpp. AIME25 moves from 95.33 to 93.00, MATH-500 from 96.68 to 97.32, LiveCodeBench-v6 from 61.37 to 69.37. GPQA Diamond drops from 89.29 to 76.57; the authors point at OpenThoughts calibration, which is heavy on math and code. On Unsloth GGUF Qwen3-8B, Q2K average across AIME25, GPQA Diamond, and MMLU-Pro rises from 50.03 to 56.28, Q3KM from 60.52 to 61.61, same wire format.
A large piece of the scalar-versus-vector gap at 2–3 bits is an optimization gap, not a format ceiling. For anyone who needs llama.cpp or vLLM scalar kernels and wants to sit close to QTIP, GSQ is the cleanest compromise so far: symmetric group-128, drop-in kernels, and a path back into GGUF. Vector methods are still barely applied at trillion-MoE scale, so the scalar route lands first. The bill is search time: 68 hours for Llama-3.1-70B on 8×H200, 10 hours for 8B, 24 hours for Kimi-K2.5 because only experts are quantized and epochs are fewer.
Auxiliary logits cost 2–5× weight memory during training, so full-model joint optimization is off the table; the method has to be block-wise or expert-wise. Local shift assumes GPTQ or GGUF initialization is already nearby; a bad start may not recover. Vector quantizers remain more expressive at a fixed bit-width, and they can still win when codebook lookup is acceptable. The Kimi GPQA drop shows calibration mix is written into the quantizer. The authors' LiveCodeBench baseline sits below the model card, and they flag the protocol mismatch. One 8B downproj stays in full precision because it was unstable, and reported bit/param excludes unquantized tensors.