Alibaba papers fix looped Transformers' compute redundancy, cutting 33% params while gaining accuracy

量子位 · wechat · 2026-09-05

Following the GPT-6 Astra leak, recurrent depth (reusing the same Transformer layers repeatedly) has exploded in popularity — along with safety concerns over unmonitorable hidden-state reasoning that OpenAI's chief scientist Jakub Pachocki had to address. The core blocker is "compute redundancy": later loops barely update hidden states.

Two Alibaba papers tackle this:

MeSH (ICLR 2026) — diagnoses loop idling (shrinking updates, high CKA similarity, low-rank collapse) and adds a Memory Buffer with per-loop, per-token read/write routers. Result: 33% fewer non-embedding params on Pythia-1.4B, yet zero-shot accuracy rises from 49.50% to 50.56%, with only 0.014% extra compute.

SpiralFormer (EMNLP 2026) — varies sequence resolution per loop, compressing tokens from 1/8 length up to full (coarse-to-fine), with right-shift to prevent future leakage. Probing confirms real global-to-local attention specialization. Compute drops from 14.08 to 13.13 TFLOPs while 5-shot accuracy improves from 51.93% to 54.37%; optimal loop ratio sits at 30–40% on a U-shaped curve.

Together they add the missing ingredient — division of labor across loops — strengthening the case for looped architectures as an efficient-scaling route beyond stacking parameters.

Original post →

Alibaba papers fix looped Transformers' compute redundancy, cutting 33% params while gaining accuracy

More from Models