PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States
Patrick Haller, Filipe Laitenberger, Alan Akbik
cs.CL
2026-10-21
PACE quantizes linear-attention recurrent states to 4 bits by rounding once every 8 tokens, within a point of 8-bit DAMP on three larger models at two thirds of the memory.
Linear-attention models such as the Qwen3.5 hybrids replace the Transformer KV cache with a recurrent state of fixed size. The state does not grow with context, but a server holds one copy per concurrent sequence, and at 64 concurrent sequences fp32 states take 10–41% of the memory that weights and states occupy together. Cutting the state to lower precision buys concurrency on the same accelerator.
Existing state quantizers (Quamba2, Q-Mamba, DAMP) all round the state after every generated token, and all report accuracy loss at four bits; the usual remedies are raising the width to eight bits or fine-tuning some weights. A NVIDIA report tried rounding less often but only at eight bits and above. The overlooked fact is that prefill already works differently: chunked kernels process 64 to 1,024 tokens at a time and write the state once per chunk, so every published quantizer already rounds sparsely during prefill. Per-token rounding in decoding is an artifact of kernel granularity, not a mathematical requirement.
PACE is a post-hoc quantizer: no training, no fine-tuning. Three components, each tied to a finding.
Round once every eight tokens. Each rounding injects error into the state, and the recurrence's decay shrinks old error over time. The paper bounds the retained error: the energy of one rounding survives to the next as ah to the power 2dP (ah is the head's mean decay, dP the rounding period), giving an accumulation bound of eh/(1−ah^2dP). Longer periods leave more decay between roundings, so less error survives. Tokens generated in between go into a small decode buffer of recent keys and values, and the state is advanced over the buffered tokens in one chunked step. Serving work such as KVBuffer and DeltaLog already uses these buffers for speed; PACE reuses them to control rounding frequency. The memory accounting is honest: resident bits per value folds the buffer cost into the per-value bit count, and all comparisons use that metric.
Channel-aligned scale groups. In group quantization, entries sharing a scale inherit the largest entry's step size, so one outlier coarsens the whole group. The large entries of the gated-delta state cluster in columns, all at the same value coordinate. The cause sits in the delta value, the part of each token actually written into the state: some of its channels stay large across many tokens, and the outer-product accumulation piles them into one column. The check is clean: permute those channels at every token and the columns vanish; permute the key entries and the columns get stronger. PACE cuts groups along the key axis, each group drawn from a single column, where DAMP cuts along the value axis and lets one step size span columns of very different magnitude.
Head-level precision allocation. Heads with decay close to 1 hold rounding error longest. PACE ranks heads per layer by the retained-error bound and keeps the worst quarter at eight bits, the rest at four. The ranking is computed once per checkpoint from eight held-out 2,048-token windows, putting the payload at 5.66 bits per value, level with DAMP INT4.
Two variants: PACE-4 keeps every head at four bits (5.03 resident bits); PACE-4/8 adds head allocation, an integer-zero-point grid and a mixed buffer (6.82 resident bits, 0.69 of DAMP INT8's 9.9).
Five benchmarks (GSM8K, MMLU-Pro, IFEval, RULER-32k, MATH-500) averaged over four backbones (Qwen3.5-4B/9B/27B, Qwen3.6-35B-A3B). Every method except DAMP INT8, which keeps its published per-token config, rounds at dP=8:
| Method | resident bits | 9B avg | 35B-A3B avg |
| Full-precision state | 32.00 | 84.5 | 86.5 |
| DAMP INT8 | 9.90 | 83.7 | 86.3 |
| DAMP INT4 | 6.44 | 79.7 | 69.0 |
| Q-Mamba DSQ | 5.03 | 72.8 | 71.2 |
| PACE-4/8 | 6.82 | 83.7 | 85.0 |
| PACE-4 | 5.03 | 80.0 | 77.3 |
On 9B, PACE-4/8 ties DAMP INT8 exactly (83.7 vs 83.7) at 0.69 of its resident memory; on 27B it is 87.3 vs 87.4, on 35B 85.0 vs 86.3. DAMP INT4 collapses on 35B (69.0 average, MATH-500 down to 43.8), where PACE-4/8 with an int8 buffer at the same 6.44 resident bits scores 78.3. PACE-4 holds the same memory as Q-Mamba DSQ and scores higher on every backbone (77.3 vs 71.2 on 35B). Caveat: that Q-Mamba is the authors' reimplementation of the DSQ quantizer without the original fine-tuning of about 2% of weights.
Ablations: a period of 8 beats per-token rounding at every width below eight bits on all four backbones, and captures most of what a period of 32 buys at a quarter of the buffer memory. Key-axis groups reach 7.719 perplexity on 4B where value-axis groups reach 8.158; head allocation at equal resident bits beats DAMP INT4, 7.687 vs 8.003. Throughput on one H100 PCIe at 512-token context: periodic four-bit decoding undercuts per-token bf16 TPOT from batch 16 up, by 18.6% at batch 128, with state-update latency 2 to 3.1 times lower; at batch 1 it is 1.4% slower. Most of the TPOT reduction comes from the buffer itself, the four-bit state adds a little more.
For anyone serving linear-attention models this is directly usable: no weight changes, no fine-tuning, just different rounding timing and group orientation, and state memory drops to about two thirds of eight-bit DAMP's. The state is the memory line item that scales with concurrency, so bits saved convert directly into sequences served per GPU. The rounding-period view travels too: any recurrence with decay has this knob, and nobody had isolated it at four bits. That said, the method assembles known parts (decode buffers, group quantization, mixed precision); the gain comes from aligning all three to one error analysis. A solid incremental improvement, honestly labeled as such.
The authors list their own: only four Qwen3.5/3.6 hybrid models, all gated delta rule; the column structure, head-decay statistics and head ranking are measured only on these models and not shown to transfer to other families, fully recurrent models, or other update rules. The prefill period is fixed at the kernel's 256-token chunk and never varied. Throughput is timed only for a uniform four-bit state, not for PACE's key-axis groups or head allocation. Static scales, stochastic rounding and rotations are untested under periodic rounding.
Reading the tables adds more. On the 4B model PACE-4/8 averages 79.6 against DAMP INT8's 80.8, so the advantage does not hold at the smallest size. PACE-4/8 trails DAMP INT8 on IFEval at all four sizes (81.3 vs 83.4 on 9B); instruction following is its consistent weak spot. The RULER-32k gap barely moves across decode periods, which the authors attribute to prefill, so longer periods will not close that part of the loss.