Nanbeige open-sources Depth Hybrid Attention, beating full attention in hybrid LLM architectures
syedshad · reddit · 2026-11-03
Nanbeige (面壁智能) released Depth Hybrid Attention (DHA), an open-source architecture research that adds depth-wise value mixing to hybrid attention models, letting each layer selectively reuse representations from earlier depths with little extra compute or KV-cache overhead.
- Two variants: Depth Window Attention (DWA) for GQA+SWA and Depth Delta Attention (DDA) for MLA+KDA
- Both outperform full-attention and hybrid-attention baselines on loss curves, short-context benchmarks, and long-context retrieval while keeping hybrid attention's low KV-cache footprint
- Per-token depth bank stores KV anchors; a depth router reads and mixes them before temporal attention
Code is available on GitHub.
Related event: Nanbeige Open-Sources Depth Hybrid Attention, Beating Full Attention(2 posts)→
More from Research
- SAIL AI-Tx Challenge finals open to all teams: $12.5K for rare disease AI treatment reports — zakkohane · 2026-11-04
- Terence Tao predicts an explosion of experimental mathematics as AI maps math difficulty — haider1 · 2026-11-04
- Google DeepMind Genomics team hiring 2027 Student Researcher with RL and post-training skills — alexechu_ · 2026-11-04
- SAIL AI Challenge opens final phase: turn rare disease diagnoses into treatment reports — zakkohane · 2026-11-04
- qwenFS and qwenGPU: a filesystem and SIMT GPU compiled into Qwen3.5-2B's weights — CatAstro_Piyush · 2026-11-04
- D&A turns any VLM into a zero-shot robot reward model with SOTA on three benchmarks — KyleMorgenstein · 2026-11-04