Alibaba’s SpecFormer softens embedding spectra to fight attention collapse in recommendation
_reachsumit · x · 2026-07-28
Alibaba's SpecFormer tackles a recommendation-system failure mode the authors call embedding and attention collapse. They argue that directly applying standard Transformers to recommendation often underperforms simpler models because long-tail, heterogeneous recommendation data causes the singular-value spectrum of embeddings to collapse around a few dominant directions.
The paper proposes three mechanisms: a learnable spectral softening module to smooth the singular-value distribution of input embeddings, a spectrum-softened attention module, and a spectral residual position encoding. The goal is to keep the representation space from collapsing so the Transformer can scale better with depth in recommendation settings.
More from Research
- MIT's Neil Thompson: AI comes for tasks before jobs, and fixed costs favor big firms — bigdata · 2026-10-16
- UMD's PAIR framework bridges perception and action in VLA models via bridge tokens — Kaixi Feng · 2026-10-16
- CI-MSE offline metric hits -0.87 Spearman correlation with real robot task success vs weak raw MSE — chris_j_paxton · 2026-10-16
- Colossal Biosciences uses AI to decode genetic switches behind the extinct dodo — castrotech · 2026-10-16
- Colossal Biosciences uses AI to identify gene switches in dodo de-extinction push — castrotech · 2026-10-16
- Cheng Lou open-sources Handwired: hand-written weights turn code into training-free neural nets — dotey · 2026-10-16