2026-09-23
PayPal's XMerge: quiet-layer select plus local re-fit. Dropping 4 layers, CORE first on 6/7 0.5B-8B models, zero collapses in 14 cells, no extra inference params.
Decode is sequential in transformer layers, so depth turns directly into latency and KV-cache traffic. Dropping whole blocks keeps the serving interface: hidden size, attention, vocabulary, and the stack stay standard, and a shallower transformer runs on stock infrastructure. Width pruning changes tensor shapes. Unstructured sparsity needs special kernels. Depth compression avoids both.
Quality loss is uneven, and prior methods disagree on which layer to cut and how the neighbor should absorb it: drop, average, analytic fold, or insert a new module. PayPal AI writes every depth-compression method as two steps, \(C = M \circ S\). The selector \(S\) chooses the block; the operator \(M\) decides how an adjacent block absorbs it. Failures split the same way: removing a block that mattered, or failing to keep the two-block map of a safe pair.
XMerge pins both decisions down.
The selector is label-free. Relative magnitude (RM) scores how far a layer pushes the hidden state. The angular axis reuses ShortGPT's Block-Influence (BI), scoring directional change. Each axis is standardized across eligible layers; the score is \(sl = \max(zl^{RM}, zl^{BI})\), and the smallest wins. Max, not a mean, so a quiet axis cannot hide a loud one, and there is no mixing weight. A block is removable only when both axes are quiet.
Merging keeps one existing standard block from the adjacent pair and re-fits all of its parameters (attention, MLP, norms) for 300 Adam steps so the block matches the original two-block output boundary. The loss is MSE on WikiText-2 train activations. Hyperparameters are frozen across models: learning rate \(10^{-5}\), batch 16, 128 sequences of 512 tokens. No extra inference parameters, no end-to-end fine-tune. Turn the 300 steps off and the method becomes a selected drop that falls below published baselines. The gain is reconstruction, not selection.
For \(k>1\), selection order is computed once on the dense model; reconstruction targets refresh after each merge. Cross-axis fusion is byte-identical to RM on five of seven backbones and only changes the layer set on Qwen3-0.6B and Qwen3-1.7B, where RM and BI are nearly uncorrelated.
Seven backbones (Llama-3.2-1B/3B, Llama-3-8B, Qwen2.5-0.5B, Qwen3-0.6B/1.7B/8B) against ShortGPT, LaCo, MKA, SWM, and CoMe, at \(k \in \{1,2,4\}\). CORE, a 22-task centered aggregate, is primary; zero-shot MMLU and WikiText-2 test perplexity are secondary. Setting A compares constructed checkpoints with no follow-on recovery.
At \(k=4\), XMerge ranks first on six of seven backbones on CORE and, separately, on six of seven on MMLU (five of seven on both). Mean CORE retention versus dense is 69% at \(k=4\), against about 60% for the best competing merge and 54% for ShortGPT. At \(k=1\) both XMerge and ShortGPT keep about 94% of dense CORE: mild compression is a close fight, and the gap opens under heavier removal.
| Model | XMerge CORE | Best baseline |
| Llama-3.2-1B | 0.201 | MKA 0.130 |
| Llama-3.2-3B | 0.406 | ShortGPT 0.373 |
| Llama-3-8B | 0.534 | ShortGPT 0.528 |
| Qwen2.5-0.5B | 0.178 | ShortGPT 0.168 |
| Qwen3-0.6B | 0.179 | MKA 0.165 |
| Qwen3-1.7B | 0.271 | MKA 0.289 |
| Qwen3-8B | 0.310 | SWM 0.262 |
A task-level bootstrap puts 95% CIs that exclude zero on the three largest CORE margins. The narrow wins and the Qwen3-1.7B loss all include zero and should be read as ties. The six-of-seven sign test is \(p=0.06\). On perplexity, only XMerge and LaCo avoid blow-ups in the \(10^2\)–\(10^4\) range, and XMerge is lower than LaCo on six of seven.
Across 14 (model × zero-shot/ICL) cells, XMerge is the only operator that never collapses (centered CORE never below 0.10). Its worst cell is still +0.172, and it is top-2 in both regimes on all seven models. CoMe breaks on 7 of 14. With selection fixed, reconstruction lifts Llama-3.2-3B from 0.344 to 0.406 and Qwen3-0.6B from 0.136 to 0.184. Calibration was probed once, on Llama-3-8B at \(k=4\): ECE rises +0.010 over dense 0.125, versus +0.110 for MKA.
Batch-1 decode speedup at matched depth is method-independent, about 1.04/1.09/1.17× at \(k=1/2/4\). Construction spends extra gradient time, minutes to 4.4 hours, and the measured speedup recovers that cost after roughly 1.9k–24k requests of 256 tokens.
If the constraint is stock serving plus fewer sequential layers, the output is an ordinary \(L-k\) transformer: no sparse kernels, no new modules. Calibration uses WikiText-2, not downstream labels.
This is an incremental operator, and the increment shows up mainly when more layers come out. At \(k=1\) ShortGPT often ties or wins. Local label-free reconstruction is closer to post-training quantization than to end-to-end recovery. The 8B cap is a 40 GB GPU-partition budget, not a limit of the method.
A single compression seed (42). The task bootstrap bounds noise over 22 tasks, not reconstruction-seed variance. Coverage is dense decoder-only Llama/Qwen from 0.5B to 8B: no MoE, encoder–decoder, or multimodal. WikiText-2 test perplexity is partly in-domain with the calibration split. The merge-direction heuristic \(S{patch}\) is not ablated.
XMerge is not universally best: it loses \(k=4\) CORE to MKA on Qwen3-1.7B, and it is not lowest-perplexity on Qwen3-8B. Because the objective is hidden-state MSE to the dense model, CORE, MMLU, and perplexity partly measure the same proximity; three agreeing metrics are not three independent confirmations. Calibration is one backbone; hallucination and refusal are untested. Setting A withholds the mergers' native LoRA/distillation, so it is not a bake-off against each paper's full pipeline.