MACaQ keeps Hadamard transforms online, per layer: 11x faster W4A4 quantization at 2.3% overhead

MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations

Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli

cs.LG

2026-11-07

Per-layer MXFP4 W4A4 quantization with the basis transform kept online: 2.3% inference overhead, 11x faster calibration at equal or lower perplexity, 122B model quantized in 64 min.

What problem this solves

W4A4 quantization, weights and activations both at 4 bits, is the main route to cheaper LLM inference. The obstacle is a handful of activation channels with far larger magnitudes than the rest; naive round-to-nearest quantization breaks on them. The standard fix since QuaRot is a change of basis: a Hadamard rotation spreads the outlier energy across channels, and per-channel scales balance quantization difficulty between weights and activations.

The friction is the activation-side transform. Computing it at inference costs time, so prior work (QuaRot, DuQuant++, FlatQuant) propagates it upstream and absorbs it into the previous layer's normalization parameters or adjacent projections. That couples layers together: touching one linear layer can force edits to norms, residual branches, or attention projections. Every new architecture needs bespoke engineering, some architectures have nowhere to absorb the transform, and quantization slows down under the cross-layer bookkeeping.

MACaQ, from Huawei's Computing Systems Lab in Switzerland, takes the opposite bet: keep the transform inside the layer, run it online, and pay the small cost.

Method

MACaQ is a post-training method that quantizes each linear layer in isolation, with the same objective as GPTQ: minimize that layer's output reconstruction error. Four steps per layer:

Scaling precedes rotation because importance weighting must happen in the unrotated domain. Quantization itself streams: after hooks collect each layer's Hessian, layers are processed one at a time, with only the current layer on the GPU and the remaining weights in host memory. The overhead has a closed-form bound as well: online transform operations over matmul operations equal 3/m at block size 32, under 0.3% for the smallest linear layer (m = 1024) in Llama-3-8B.

Results

Llama-3-8B, MXFP4 W4A4:

MethodWiki2 pplC4 ppl
FP166.149.46
MR-GPTQ7.1311.37
DuQuant++ (GPTQ)7.1711.66
MACaQ6.908.61

MACaQ reaches 6.90 in about 150 seconds; sweeping calibration tokens and flip budget, it matches the best DuQuant++/MR-GPTQ quality in 11x less quantization time, and posts the lowest perplexity among quantized methods on Qwen3-32B and Qwen3.5-35B-A3B too. Generative results split narrowly: on Qwen3-32B, IFEval 83.20% against 83.60% for FP16, and MATH-500 97.26% where MR-GPTQ edges ahead at 97.44%; on Qwen3.5-35B-A3B, IFEval 85.42% sits slightly above FP16's 85.29%.

Throughput: Qwen3-32B in vLLM decodes at 69.8 tok/s at batch 1 versus 22.8 for FP16 (3.06x), and 5252.8 versus 3061.5 tok/s at batch 256. The online-transform overhead shrinks with model size, from about 8% on Qwen3-0.6B to 2.3% on Qwen3-32B. In compute-bound kernel comparisons MACaQ also passes Marlin.

Two generality experiments carry the paper. FLUX.2-klein-4B, an image diffusion model, is quantized with zero architecture-specific changes and lands closer to the BF16 original than the official NVFP4 release on PSNR (15.04 vs 14.85), LPIPS (0.218 vs 0.235) and DINOv3 similarity, at fewer bits per weight (4.25 vs 4.5) and lower VRAM (13.1 vs 13.3 GB). Qwen3.5-122B-A10B carries about 244 GB of FP16 weights; MACaQ streams it through a single 96 GB GPU at 32.5 GB peak and finishes in 64 minutes, with WikiText-2 ppl 5.14 against 5.06 for FP16. The 74 GB output runs on the same card. Ablations show the four components contributing comparably: RTN 8.28, exponent search 8.09, rotation 7.65, online scale 7.08, Flip-Refinement 6.87.

Why it matters

The paper removes a default assumption of the field: that basis transforms must be propagated across layers to be affordable. For 2.3% overhead you get architecture independence (the diffusion result is the evidence), layer-at-a-time streaming (a 122B model quantized on one 96 GB card), and coverage of every linear layer, including those with no absorption target. Where GPTQ-style methods assign weights in a fixed sequential order, Flip-Refinement starts from RTN and revisits codes in parallel across rows; that is where the 11x comes from. Triton kernels are already in vLLM and the code is open, so the practical barrier to W4A4 deployment is low.

Limitations

The authors' own list: the overhead analysis counts arithmetic operations, while real numbers depend on kernel fusion and memory traffic; small models do not amortize the online transforms (about 8% at 0.6B); quality plateaus around 2M calibration tokens and 512 flips, and the default configuration (200k tokens, K = 128) trades quality for time.

Questions the paper leaves open: the C4 perplexity of 8.61 on Llama-3-8B sits below FP16's 9.46 and nearly three points ahead of the strongest baseline, with no mechanism offered for a quantized model beating full precision on out-of-distribution text; that number deserves a second look. α and λ were tuned on Llama-3-8B with WikiText, and there is no ablation of how they transfer to other model families or calibration sets. MoE models get only their expert layers quantized, with embeddings and lm-head left in high precision. Baseline runtimes were re-measured by the authors on shared hardware, so the 11x figure depends on whether each baseline was tuned to its own optimum.

Terms

Source

What people are saying

Related papers

All paper explainers