G$^2$PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation
Ruikang Liu, Haoli Bai, Yuxuan Sun, Qian Zhang, Wenzheng Cai, Yanqi Hao, Feiyu Wang, Weidong Zhong, Zhuang Wang, Tong Yang, Xiangsheng Zhou
cs.CL, cs.AI
2026-09-25
G²PTQ refreshes exact gradients and Hessians per Transformer block, with trust-region scaling. On W2A16 it halves KL vs GPTAQ (1.83 to 0.90) and lifts average QA by 6.45 points.
GPTQ treats quantization as second-order compensation. After a weight is rounded to low bits, the rest of that row is updated with the Hessian so the layer output stays close. That recipe is the default PTQ path for LLMs. Two gaps sit next to each other.
Matching each linear layer with MSE does not keep the full-model predictive distribution in place. OAC, GuidedQuant, and YAQA switch the objective to end-to-end NLL or KL and approximate the Hessian with Fisher information, but they compute that Hessian once at the start. Later weights move, and the curvature they follow goes stale. FOEM puts the discarded first-order term back, yet it builds the gradient from a first-order Taylor expansion that is exact only under layer-wise MSE. On block-wise MSE or KL the approximation is biased.
Global objectives drop the gradient. First-order methods stay local. The two lines never met.
G²PTQ moves the objective from a single linear layer to a Transformer block. Intermediate blocks minimize block-wise MSE between quantized and full-precision hidden states. The last block minimizes KL between the two predictive distributions. Before each block is quantized, one backward pass through that block only refreshes both gradient and Hessian. Cheaper than a full-model backward, and not frozen from step zero.
Exact Hessians of nonlinear blocks are intractable, so both losses are approximated by gradient outer products: Generalized Gauss-Newton with Gaussian noise on the hidden state for MSE, Fisher with labels sampled from the full-precision model for KL. Output channels are split into g groups that share a Hessian; the main runs use g=4, following GuidedQuant.
The first-order term is the exact backprop gradient, not a Taylor stand-in. That exact step is scale-sensitive: if the two losses are multiplied by a and b, the update norm scales as a²b⁻⁴ and can explode. A trust-region factor β in [0, 1] shrinks the gradient step so the first-order loss change stays within α times the GPTQ term. α is grid-searched per block on [0, 3] with step 0.1, then reused across bit-widths and sibling models. On an 8B model that transfer speeds quantization by 1.20x to 1.33x.
The matrix that holds gradient times inverse Hessian is updated recursively from a Cholesky factor, bringing complexity back to O(max(dcol³, drow · dcol²)), the same order as GPTQ, with the usual lazy-batch. G²PTQ splits each block into four stages and refreshes before every stage.
Evaluation covers 13 dense models from 0.6B to 70B and two MoEs up to 125B. Weights use symmetric per-channel quantization, activations symmetric per-token. Dense models are calibrated on 1,024 sequences of length 2,048; MoEs use 8,192. All methods apply a Hadamard rotation first. Baselines are RTN, GPTQ, GuidedQuant, and GPTAQ.
Averaged over three models at W2A16, G²PTQ cuts KL versus GPTAQ from 1.83 to 0.90 and raises QA by 6.45 points. The 2-bit setting is where the gap is largest.
| Setting | Baseline | G²PTQ / G²PTQ |
| Qwen3-0.6B W3A16 QA | GPTQ 33.09 | 39.83 |
| LLaMA3-8B W2A16 QA | GPTAQ 42.38 | 51.16 |
| LLaMA3-70B W4A4 QA | GPTAQ 68.38 | G²PTQ 72.94 (+4.56) |
| Qwen3.8-Flash-Next 125B W4A16 QA | BF16 78.28 | G²PTQ 77.95 (drop 0.33) |
Ablation on Qwen3-0.6B at 3-bit: block-wise alignment lifts QA from 33.09 to 37.40. Refreshing the statistics, adding the exact gradient, then putting KL on the last block reaches 39.83. Remove trust-region scaling and PPL hits 1.29×10¹², QA falls to 28.97. In the same setup, the exact gradient beats FOEM-style Taylor approximation by 2.09 QA points (39.83 vs 37.74).
Appendix totals: lowest average KL in 38 of 39 weight-only settings; highest QA on 12 of 13 dense models at 2-bit. On 4-bit weight-activation for seven mid-size dense models, every model records the best QA.
Calibration is more expensive. An 8B model on one accelerator takes G²PTQ 12.54 GB and 2.24 hours, against GPTQ at 4.62 GB and 0.43 hours. The 125B MoE on eight accelerators takes G²PTQ 86.72 GB per device and 20.30 hours, against GPTQ at 19.48 GB and 7.98 hours.
This is a merge inside the GPTQ family: block-level supervision, a Hessian that no longer goes stale, and the missing first-order term, all in one update. Pipelines already running GPTQ or GPTAQ can swap in the new compensator without QAT. Complexity is claimed to match GPTQ. Code is public.
Gains concentrate where error is large: 2-bit weights, and 4-bit weights plus activations. At 4-bit weight-only the gap to GuidedQuant is already small; that part is incremental. α can be searched once and transferred, so a new bit-width does not need a fresh grid search. Hessian-guided clipping (H-Scale) adds 2.12 QA points on Qwen3-4B at 3-bit, and does nothing or slightly hurts Qwen3.5. The main experiments keep weight-MSE clipping.
There is no standalone Limitations section. The costs and open ends are still visible.
Calibration is clearly heavier than GPTQ: about 3× memory and 5× time on 8B, and 20 hours on a 125B MoE across eight accelerators. G²PTQ also wants a larger calibration set than GPTQ. The Hessian approximations (GGN / Fisher, near-independent channels, sparse causal attention) are not audited for approximation error on their own. The main tables omit FOEM, so the first-order gain is only shown in the authors' own ablation on Qwen3-0.6B.
The W4A4 writeup already says activation outliers dominate, so weight updates only compensate indirectly. After noting that H-Scale is model-dependent, the choice of clipper is left to future work. The 38/39 and 12/13 counts are appendix summaries; the main tables only show three dense models in full.