Apple-style compression: LSP learns which subspaces to drop, cutting LLM weights 70%

Massimo Bini · hf · 2026-10-08

A new paper introduces Learnable Subspace Projections (LSP), which learns low-rank subspaces to discard end-to-end instead of using local closed-form criteria that ignore error propagation through depth.

Each linear layer (or tied group sharing activations) gets an orthogonal projector, jointly optimized against a global objective (KL to the dense model's outputs) with pretrained weights frozen; projectors merge into standard low-rank factors afterward.
In attention, a single narrow shared latent can replace full K/V, shrinking combined weights+KV cache memory 13.5x at 128k context (vs 6.5x max for untied baselines).
At 70% compression, Llama-2-7B reaches 10.9 WikiText-2 perplexity and 42.2% zero-shot accuracy vs 13.3 and 36.0% for the strongest baseline; factorized models decode up to 1.6x faster at small batch sizes.

Original post →

Apple-style compression: LSP learns which subspaces to drop, cutting LLM weights 70%

More from Infra