New Relation architecture replaces Attention with explicit Self/Exchange relation decomposition
青稞AI · wechat · 2026-09-04
The author documents two papers: MemGLU, testing whether SwiGLU's open positive tail is necessary via a closed-tail gating function (which drew a code request from SELU author Günter Klambauer within a day of arXiv posting), and the main work, "Ask Self, Ask Others: Relation Is All You Need."
Key idea: split Attention's fused relation-formation and information-routing into separate steps. Token relations are decomposed into diagonal Self and off-diagonal Exchange roles, projected by a pair of nonlinear functions chosen after 23 ablations, with a log correction term fixing Exchange's natural growth with sequence length. The multi-head version uses alternating Givens rotations to decouple relations from information across heads. Because the relation matrix is explicit, the author derives a FlashRelation blockwise computation path needing only running statistics — opening a route toward Linear variants.
More from Research
- Frontier Math Will Need Billions Like Big Physics, Say Researchers on 'Big Math' Era — sytelus · 2026-09-23
- Cohere Labs session probes scaling law reliability and offers a research checklist — Cohere_Labs · 2026-09-23
- SoL-Pi paper: auto-research loops save $4-13 per hour on coding agents — alex_verem · 2026-09-23
- NVIDIA's SoL-Pi lets AI rewrite agent harnesses, cutting tokens 44.7-49% — alex_verem · 2026-09-23
- Slingshot RL framework jailbreaks Qwen2.5-32B at 67% success, transfers zero-shot to Gemini 2.5 Flash — j_foerst · 2026-09-23
- TLAPS-Bench launches in alpha to test whether AI agents can formally prove system correctness — tianyin_xu · 2026-09-23