GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression
Baher Mohammad, Ammar Ali, Stamatios Lefkimmiatis
cs.LG
2026-09-22
GeoPair shares a dictionary across geometry-matched layers, no fine-tuning. Llama-3 8B at 20% compression: average accuracy 70.36 to 68.17.
Post-training factorization already approximates each linear projection in a whitened activation space. The default is still one layer at a time. Deep transformers reuse directions across depth, so independent compression stores the same basis twice.
Cross-layer sharing exists, most visibly in Basis Sharing: glue adjacent layers together and average their covariances into one global whitening. Averaging erases per-layer activation geometry. Fixed adjacency also misses non-local matches, including attention and FFN projections that happen to align. GeoPair treats both the pairing and the shared-dictionary update as optimization problems, with no fine-tuning.
The objective is activation reconstruction on a small calibration set, not a raw Frobenius fit on weights. Each layer gets its own Cholesky whitening from a Tikhonov-regularized Gram matrix, so two layers do not live in the same metric. Two weights \(W1,W2\) share a dictionary \(D\) and keep private coefficients \(C1,C2\). The loss is the sum of whitened reconstruction errors.
Alternating minimization has closed forms:
Pairing is global. Every projection that shares an input dimension, attention mixed with FFN, becomes a vertex of a complete graph. Edge weights are a scale-normalized Frobenius submatrix distance in each layer's whitened space. Edmonds' Blossom computes a maximum-weight matching. The dictionary is initialized from an SVD of the concatenated weights.
Calibration uses 256 FineWeb sequences. Sparse runs set the KS ratio (dictionary atoms over nonzeros per coefficient column) to 2.5.
Llama-3 8B at 20% compression, zero-shot average over PIQA, HellaSwag, LAMBADA, ARC, SciQ, RACE, and MMLU:
| Method | Avg acc | WikiText PPL |
| Uncompressed | 70.36 | 7.26 |
| Basis Sharing | 53.47 | 41.26 |
| Sylvester only | 56.4 | 26.82 |
| optimal grouping | 57.46 | 24.56 |
| Full GeoPair | 68.17 | 9.43 |
On the 1B model at the same ratio, the full pipeline moves 57.61 back to 54.09. Basis Sharing falls to 33.11 and WikiText PPL jumps from 11.60 to 928. Dropping optimal grouping from the sparse pipeline cuts average accuracy from 54.09 to 53.06.
Against pruning methods on Llama-2 7B at 20% compression, GeoPair scores 63.58 with no recovery training, above WANDA at 57.23 and slightly above LoRAShear at 62.22, which does fine-tune. The uncompressed average is 68.59. Qwen3-8B, Gemma-3 12B, Phi-4 14B, and Qwen3-32B beat Basis Sharing from ratio 0.2 to 0.4. Phi-4 14B rises from 72.09 to 74.07 at 20% compression; the paper calls that small-rank denoising. On Wan2.2 5B video, X-CLIP over 50 prompts is 0.2165 uncompressed, 0.2164 at 20%, 0.2111 at 40%.
This is compression for deployment: no healing set, no extra LoRA. Anyone already doing data-aware low-rank or dictionary compression can swap two pieces, stop averaging covariances, and stop pairing only neighbors. The video run shows the same pipeline can touch a DiT, but a \(10^{-4}\) CLIP gap on 50 prompts is a sanity check.
A 2.2-point drop at 20% on 8B is a usable incremental gain. At 40% on Qwen3-8B, average accuracy falls from 70.46 to 58.88. That is no longer near-lossless.
The authors restrict sharing to pairs. Three or more layers would need a multi-term Sylvester system that does not diagonalize in closed form. Iterative or tensor substitutes are left as future work.
The pruning table uses a different benchmark mix than Table 1, copied from the pruning papers, so 63.58 is not comparable to 68.17. On Gemma-3 12B at ratio 0.3, Basis Sharing's LAMBADA PPL hits 164 against GeoPair's 4.39; that looks like a collapsed baseline. Switching calibration among WikiText, Alpaca, and FineWeb moves 1B / 40% average accuracy by a point or two, but LAMBADA PPL can jump from 21.69 to 39. Growing the calibration set from 256 to 1024 adds only 0.9 accuracy points. The paper reports parameter ratios, not wall-clock decode speedup.