New Top-A Attention Breaks Diagonal Limits of Multi-Head Graph Attention
rishabh16_ · x · 2026-10-10
A new arXiv preprint, "Let the Heads Talk: Beyond Diagonal Graph Attention," connects sheaf neural networks and multi-head attention. The authors show standard multi-head attention implements only diagonal edge maps—each source head can only feed the corresponding receiver head.
- They isolate the transport primitive via quiver representations and propose Topological Attention (Top-A), which learns edge-dependent off-diagonal routes enabling cross-head communication before neighborhood aggregation
- They prove this operation cannot be absorbed into a single shared linear map post-aggregation, and Top-A exactly recovers vanilla attention when the extra routing vanishes
- Evaluated on relational reasoning, heterogeneous graph learning, and algorithmic reasoning (including OOD generalization), showing cross-head transport helps most on interaction-heavy tasks
More from Research
- Embedding arithmetic demo: cartoon − animation − comic = diagram — mayfer · 2026-10-23
- Omnii Genome: Single Generalist Model Outperforms AlphaGenome Across All Benchmarks — JosephJacks_ · 2026-10-23
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23
- Word Arithmetic on Embeddings: "Bathroom Minus Toilet Equals Kitchen" Beats SAE Probing — mayfer · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23