TPA: Tensor Product Attention unifies MHA/GQA/MLA, NeurIPS 2025 Spotlight
burny_tech · x · 2026-09-18
The thread highlights the arXiv paper "Tensor Product Attention Is All You Need" (NeurIPS 2025 Spotlight). Key points:
- TPA uses tensor decompositions to represent queries, keys and values compactly, substantially shrinking KV cache at inference;
- Seamlessly integrates with RoPE, improving quality alongside memory efficiency;
- The proposed T6 architecture surpasses or matches MHA, MQA, GQA and MLA baselines on perplexity and standard benchmarks;
- Decoding-stage efficiency enables longer sequences under fixed resources.
The quote tweet notes MHA, GQA and MLA can all be seen as special cases of a broader tensor-factorization attention (Tucker Attention), fully compatible with FlashAttention.
More from Research
- Who Gets Credit When AI Proves Collatz? The New Math Attribution Dilemma — jd_pressman · 2026-09-18
- SceneAgent: agentic pipeline turns 3D captures into physics-ready scenes for robot training — hankyang94 · 2026-09-18
- LAX launches to bridge natural math language and Lean, but looks a lot like existing tool span — lpachter · 2026-09-18
- Fari Research paper: misaligned AI may just persuade its human overseers — DG_Rand · 2026-09-18
- Biomedical world models: a framework for virtual drug trials, intervention design and planning — marinkazitnik · 2026-09-18
- LeanReact 0.1: expressing composable, provably correct React components in Lean — hargup13 · 2026-09-18