ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
Şuayp Talha Kocabay, Talha Rüzgar Akkuş, Kamer Ali Yuksel
ACL Rolling Review (ARR)
cs.CL, cs.LG
2026-08-03
ARCHead compresses the BF16 LM-head that quantizers leave untouched, reaching 1.007 relative perplexity on Qwen3-8B at 26% storage, versus 1.15 for storage-matched INT4.
Weight-only quantization compresses transformer blocks to 4 bits, but practical backends like AWQ and bitsandbytes keep the final LM-head in BF16 or FP16. The head is too sensitive: naive INT4 perturbs the vocabulary logit distribution heavily. For large-vocabulary models like Qwen3-8B-Base that BF16 head alone is about 1.18 GB, the largest slab left in a quantized deployment.
The subtlety is that naive low-bit quantization minimizes weight-space reconstruction error, when what actually matters is the output error under the real activation distribution. Those two objectives are not the same.
ARCHead splits the dense output head into three packed components and stores no dense BF16 head at all. A quantized low-rank core combines a rank-r approximation with group-wise signed INT4 residuals. An activation-derived transform, built from a damped activation covariance, weights error directions by how often real activations land on them: directions the hidden state actually hits are the ones where error leaks into the logits. A low-rank correction is fitted by truncated SVD in the transformed space and mapped back, stored as a pair of INT8 matrices. The final output adds this correction to the quantized core product.
The authors prove conditional optimality: for a fixed core and invertible transform, the unquantized correction factors minimize the activation-metric objective before factor quantization. The design is not ad hoc.
Head-only quality on Qwen3-8B-Base, WikiText-103:
| Method | Head storage | Rel. PPL |
| Dense BF16 | 100% | 1.000 |
| Row INT8 | 50% | 1.017 |
| Group INT4 | 26.6% | 1.151 |
| SVD8+INT4 | 27% | 1.211 |
| ARCHead | 25.6% | 1.007 |
ARCHead and group INT4 sit at almost identical storage (25.6% vs 26.6%), yet naive INT4 inflates perplexity by 15% while ARCHead is near-lossless. On top-1 token agreement, group INT4 manages 76.95% with KL 0.167; ARCHead reaches 93.05% with KL 0.011.
Paired with block quantizers (AWQ 4-bit or bitsandbytes NF4, both of which leave a BF16 head), replacing that head with ARCHead costs only 0.006 to 0.007 cross-entropy and changes throughput by under 2%: on Qwen3-8B, 4468 tokens/s for the BF16 head versus 4467 for ARCHead. Compression holds at 3.71x to 3.91x across five models. Against a GPTQ-style head, ARCHead cuts the degradation by a further 33.6% to 49.3% and builds 2.36x to 2.58x faster.
For anyone deploying large-vocabulary models with AWQ or bitsandbytes, the awkward BF16 head can finally go, with negligible quality and speed cost. ARCHead complements block quantizers rather than replacing them: it touches only the output projection, not MLP or attention.
The authors are explicit. ARCHead compresses only the output head, not the transformer body. The three downstream tasks (HellaSwag, TruthfulQA MC2, WinoGrande) are a sanity check, not proof that every capability survives. The memory claim is about serialized and load-time parameter storage, not peak GPU memory. And the gain is model-dependent: on Gemma-4-E4B the naive INT4 baseline is already strong (1.013 versus ARCHead's 1.010), so the two are within rounding distance and the advantage shrinks to a wash.