NeUQI reworks quantization init: 2-bit LLaMA-2 7B perplexity drops from 6953 to 17

NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs

Li Lin, Xinyu Hu, Xiaojun Wan

ICML 2026

cs.LG, cs.CL

2025-05-23

Beyond Min-Max init: NeUQI cuts 2-bit LLaMA-2 70B perplexity from >1e4 to 56.21, and a 0.5M-token distillation beats PV-tuning's 1B-token recipe.

What problem this solves

Running open-weight LLMs on consumer GPUs mostly goes through post-training quantization (PTQ): compress trained bfloat16 weights to int4/3/2 without retraining. Uniform quantization, where values sit on an evenly spaced grid parameterized by a scale and a zero-point, dominates deployment because mainstream hardware and inference kernels support it natively.

Recent low-bit work has concentrated on reshaping weight distributions (Hadamard rotations, channel-wise scaling). The initialization of the quantization parameters themselves still defaults to the Min-Max formula from a 2017 quantization guide: derive scale and zero-point straight from the weight maximum and minimum. That is fine at 4 bits and above. At 2-3 bits the grid has only a handful of points, the Min-Max starting point lands far from optimal, and downstream optimization struggles to recover.

This paper from Peking University's Wangxuan Institute (ICML 2026) spells out two constraints in Min-Max that everyone had inherited without examining. First, both scale and zero-point are pinned to extreme values, which pushes search-based methods into a two-dimensional grid over extreme-value scaling factors; LeanQuant's variant needs T² evaluations with T typically 2048. Second, the zero-point must be a k-bit unsigned integer, which simply deletes part of the parameter space. NeUQI drops both constraints.

Method

NeUQI keeps the GPTQ-style layer-wise loss, minimizing the squared error between quantized and original layer outputs, then applies a diagonal proxy-Hessian approximation so each row's loss becomes an importance-weighted sum of per-weight squared quantization errors.

The key move is the zero-point. With the scale fixed, the loss as a function of z is piecewise quadratic: each weight's error expression changes only when z crosses a rounding boundary, and those boundaries are known half-integer points. Enumerating every interval and summing over all n weights is quadratic-time work, hopeless at n≈2048. Two designs bring it to O(n log n):

With an efficient zero-point in hand, the two-variable problem collapses to a one-dimensional search over scale: T=2048 candidates between 0 and the Min-Max scale, a coarse pass of 64 to localize, then local refinement. The zero-point is also allowed to be fractional, costing about 0.1 extra average bits at 2-bit group-wise (2.14 to 2.25), which double quantization can offset. On 2-bit LLaMA-2 7B, initializing a single query matrix drops from 112 seconds to about 0.026 seconds, at under 0.3% extra loss.

Results

Hardest setting, 2-bit channel-wise, LLaMA-2 7B:

MethodWiki2 ppl ↓Avg acc (5 benchmarks) ↑
GPTQ695335.08
GPTAQ126934.97
MagR12939.54
NeUQI17.1447.24

The gap widens with size: at 2 bits on LLaMA-2 70B, GPTQ, GPTAQ, and MagR all blow past 10000 perplexity while NeUQI sits at 56.21 with 46.49 average accuracy. On LLaMA-3 8B, MagR reaches 387 versus 64.47 for NeUQI. On Qwen 2.5 72B, NeUQI hits 10.79 Wiki2 and 67.21 accuracy against MagR's 19.25 and 55.00.

Add a light distillation (0.5M tokens, one epoch, aligning final-layer hidden states) and 2-bit LLaMA-2 7B reaches 8.38 Wiki2 and 9.81 C4, beating PV-tuning's 8.49 and 10.78. PV-tuning trained on 1B tokens, three orders of magnitude more data. Swapping the initialization inside EfficientQAT helps too: LLaMA-3 8B at 2 bits goes from 9.80 to 8.87 Wiki2 and from 59.37 to 61.49 accuracy.

A practical 3-bit finding: at equal memory, a quantized large model beats a smaller BF16 model. Qwen 2.5 72B at 3 bits occupies 26.36 GiB versus 26.42 GiB for Qwen 2.5 14B in BF16, scoring 73.07 average accuracy versus 71.46 and 8.37 C4 perplexity versus 8.75.

Why it matters

Initialization was an inherited default nobody questioned, and this paper turns it into a reusable component: it plugs into the GPTQ pipeline, composes with Hadamard transforms (13.63 Wiki2 at W2A4 versus 20.42 for MagR), and serves as a better starting point for strong QAT methods like EfficientQAT. For local deployment the takeaway is concrete: 2-bit moves from unusable to runnable, and 3-bit is the sweet spot for trading memory into model size.

The sharper conclusion: bad initialization is hard to fix with fine-tuning. PV-tuning spent 1B tokens jointly optimizing continuous and discrete parameters and still lost to a good starting point plus 0.5M tokens of distillation. Teams still pushing low-bit uniform quantization should replace the initialization before adding anything else.

Limitations

Terms

Source

What people are saying

Related papers

All paper explainers