Rethinking Sequence Compression with Test-time Regression
Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu
cs.LG
2026-10-23
TTR summarizes token blocks as ridge-regression coefficients. RULER retrieval rises from 0.392 to 0.514 on a 345M model, and 55.6% fewer visual tokens match an MLP merger.
Block-sparse attention and multimodal models both compress tokens before the expensive part of the network. Sparse attention turns each block into a retrieval key, and the query uses that key to decide which distant blocks are worth opening. A multimodal model merges neighboring visual features into fewer tokens before they reach the language backbone. Whatever this step drops is gone.
The usual compressors ignore how tokens relate. Mean pooling weights them equally. A learned projection applies weights tied to position. Attention-weighted pooling scores each token alone and then adds those scores. Repeated background reinforces itself, and the few distinctive tokens get diluted. The math stays first-order. Pairwise similarity never enters.
TTR-Compressor replaces the weighted average with ridge regression. The fitted coefficients are the compressed vector. In a block of T tokens with dimension d, each key k is the token normalized to unit length, so only direction remains. The scalar target v = a^T x keeps the original magnitude, and a is learned. The summary h is fit so that each k predicts its v, with a diagonal penalty D whose entries are softplus(ρ). The task loss learns a and ρ together, and pretraining does not need a hand-tuned ridge coefficient λ.
Normalization is one-sided on purpose. If the targets were normalized too, the static vector a would already be a perfect fit and h would collapse toward a constant. The outer loop learns which regression is useful. The inner loop solves each block in closed form. At inference a and ρ stay fixed, and h is recomputed for the current block. Test-time here means the coefficients depend on the input. There is no gradient step at inference.
T is much smaller than d. The experiments use 4, 9, or 64 tokens, with d at least 1024. The dual form solves a T by T system with one Cholesky factorization, at cost O(T^2 d + T^3), instead of inverting a d by d matrix. The base compressor has 2d parameters. An optional convolution touches only the targets: kernel width 5 for language, 3 by 3 for vision, raising the count to 7d and 11d.
The Gram matrix writes every pairwise direction similarity into one linear system, so the effective weights are solved jointly. Content that similar tokens already explain does not get to add its full weight again. The appendix shows that another copy of the same token adds less and less. Figure 2 sets the penalty to the identity and every target to 1: three copies against one distinct token. Additive pooling gives the copies a 3-to-1 advantage. Joint regression cuts that to 3-to-2.
Test-time training and DeltaNet also regress on the input, but they maintain fast weights and read them with a query, usually through one gradient step or a delta-rule update. In TTR the coefficients themselves are the summary. Blocks do not depend on each other, so one exact solve is enough.
The language setup is HiLS-Attention with only the compressor swapped. The model has 345M parameters, 16 layers, and hidden size 1024. It trains on FineWeb at an 8K context for about 30B tokens. Blocks are 64 tokens, each query may open 32 distant blocks, and a local window of 512 stays dense. Retrieval is RULER single-needle (S-NIAH) and multi-key (MK-NIAH), from 4K to 32K.
| Compressor | RULER average |
| Sliding window | 0.0635 |
| Full attention | 0.2632 |
| Mean pooling | 0.2971 |
| Concat then project | 0.3919 |
| Attention-weighted pooling | 0.3920 |
| TTR | 0.5092 |
| TTR with convolution | 0.5144 |
Convolution beats the strongest compressor baseline by 12.2 percentage points. Plain TTR leads by 11.7. The average hides cell-level losses. On S-NIAH-1, TTR scores 0.996, 0.996, and 0.992 at 4K, 8K, and 16K, against 0.944, 0.914, and 0.876 for attention-weighted pooling. At 32K, plain TTR falls to 0.650, below that baseline's 0.740, while convolution recovers to 0.988. Full attention, trained only at 8K, scores 0 on this task at 16K and 32K. S-NIAH-2 at 32K is still only 0.126. The average of ten short-context commonsense benchmarks sits near 48.1, because those sequences never turn sparse attention on. Swapping the compressor does not wreck pretraining.
At 1B parameters and 100B tokens, the only scaled baseline is attention-weighted pooling: 0.7996 versus 0.735. TTR scores 1.000 on S-NIAH-1 at every length from 4K to 32K. The gap shows up on harder, longer cells. At 32K, S-NIAH-3 is 0.662 versus 0.550, and MK-NIAH-1 is 0.412 versus 0.314.
The vision encoder is the Qwen3-VL ViT and the language model is Qwen3-4B-Instruct. The merger trains alone for one epoch on LLaVA-558K, then with the language backbone for one epoch on 1.3M image, multi-image, and video instructions. Under 2 by 2 merging the averages are 54.93 for the MLP, 55.92 for TTR, and 56.03 with convolution. Under 3 by 3 they are 52.27, 54.30, and 54.85, a gain of 2.58 points. Text-heavy sets move the most. Against the 3 by 3 MLP, convolution gains 4.10 on OCRBench, 4.54 on InfoVQA, and 3.56 on ChartQA. Convolutional 3 by 3 averaging 54.85 sits next to the 2 by 2 MLP at 54.93, with 4/9 as many visual tokens, which is 55.6% fewer.
Dropping key normalization cuts the RULER average from 0.5092 to 0.2531. Without it, a itself is an exact fit and h is pulled toward that fixed vector. Among fixed penalties, λ = 0.001 already reaches 0.5023. A learned scalar reaches 0.4971 and a learned diagonal reaches 0.5092. λ at 0.0001 or at 1 falls to 0.4364 and 0.4238. The diagonal beats a well-chosen scalar by only about 0.0069 on average. What it buys is skipping the λ sweep, plus a separate penalty per channel. On 3 by 3 vision, a constant target scores 53.56, a linear target 54.30, and a linear target plus convolution 54.85. Deeper target heads top out at 54.55. Neighborhood context does more than a larger target head.
When T is far below d, TTR uses 2d parameters and avoids a projection that grows with d squared. On 8 H100s the learned compressors take similar wall-clock time to train. Mean pooling is the fastest and the weakest. On 16 H100s, visual TTR is slightly slower than the MLP at 2 by 2 and faster at 3 by 3. Past 16K tokens, prefill and decode for every compressor are clearly faster than full attention. Millisecond tables are not reported.
A bad retrieval key means the right block is never selected. A bad visual merge means the language model never sees the detail. The same closed-form operator beats the default compressor in both jobs.
Each block is solved exactly and differentiably, with no inner optimization loop at inference. The base compressor adds 2d parameters. The visual budget is the concrete saving: matching the 2 by 2 MLP on average score costs the 3 by 3 convolutional variant fewer than half as many visual tokens. At the same 2 by 2 rate the gain is about one point. The larger gap shows up when the merge gets more aggressive.
The main language run is a 345M model on about 30B tokens. The 1B appendix scales only attention-weighted pooling, not the other baselines, and neither run is close to a production long-context model. The swap was tested inside HiLS-Attention, which also brings hierarchical softmax, low-rank query calibration, and HoPE. Whether the same gap appears in NSA or MoBA is untested.
Full attention near zero at long context comes from an 8K training length without that extrapolation design. It is not evidence that TTR beats standard full attention. The fair comparison is among compressors inside one skeleton. Even there, S-NIAH-2 at 32K is 0.126 and multi-key hovers near 0.11. Better keys do not finish the retrieval task. On the easiest needle at 32K, plain TTR at 0.650 still loses to attention-weighted pooling at 0.740.
Vision is compared only with a concat-and-MLP merger, not with ToMe or other mergers. Each training stage is one epoch, and stage two updates the merger plus the language backbone. The TTR vector still passes through a two-layer MLP to match the language hidden size. The average can match 2 by 2 merging while OCRBench and ChartQA do not: convolutional 3 by 3 scores 60.00 and 67.16, against 62.80 and 70.64 for the 2 by 2 MLP.
The diminishing-redundancy claim comes from the Gram-matrix derivation and the toy layout in Figure 2. The paper never shows the fitted weights on real blocks. Remove normalization and the average falls to 0.25, so the operator is brittle to how it is built. The cubic term in the dual solve is cheap only for short blocks. It is not a compressor for a full long context.