Alibaba’s SpecFormer softens embedding spectra to fight attention collapse in recommendation
_reachsumit · x · 2026-07-28
Alibaba's SpecFormer tackles a recommendation-system failure mode the authors call embedding and attention collapse. They argue that directly applying standard Transformers to recommendation often underperforms simpler models because long-tail, heterogeneous recommendation data causes the singular-value spectrum of embeddings to collapse around a few dominant directions.
The paper proposes three mechanisms: a learnable spectral softening module to smooth the singular-value distribution of input embeddings, a spectrum-softened attention module, and a spectral residual position encoding. The goal is to keep the representation space from collapsing so the Transformer can scale better with depth in recommendation settings.
More from Research
- Netflix details its production LLM judge: hundreds of thousands of recommendations scored weekly — omarsar0 · 2026-08-24
- Nature Comment: Provenance, not interpretability, grounds trust in autonomous science — gabepgomes · 2026-08-24
- New Architecture RHEA: Train 1B Model on 8GB VRAM — zemondza · 2026-08-24
- Trained two 16M-param models to do generative CAD with real physics — debreuil · 2026-08-24
- Claude model helps discover complex structure on S^6, solving 60-year-old math problem — Singularitarian · 2026-08-24
- Study: Agents read instructions/notes 60.5% of the time, rarely touch API docs — dair_ai · 2026-08-24