Nanbeige's Depth Hybrid Attention beats full attention with tiny extra cost
teortaxesTex · x · 2026-11-03
Chinese team Nanbeige releases Depth Hybrid Attention (DHA), which adds depth-wise value mixing into hybrid attention architectures with minimal compute and cache overhead, letting attention layers selectively reuse KV anchors from earlier layers.
- Mechanism: each token keeps a depth bank of KV anchors from preceding layers; a depth router reads and mixes them via attention before temporal attention.
- Two variants: DWA (GQA + SWA) and DDA (MLA + KDA).
- Across loss curves, short-context benchmarks and long-context retrieval, both variants outperform their full-attention and hybrid-attention baselines while keeping hybrid attention's low KV-cache footprint.
Code is open-sourced on GitHub (Nanbeige/Depth-Hybrid-Attention).
Related event: Nanbeige Open-Sources Depth Hybrid Attention, Beating Full Attention(2 posts)→
More from Research
- Locus: autonomous discovery system adapts Boltz-2 and searches antibody sequence space with ESM — yusufroohani · 2026-11-04
- TRL adds native support for MOPD, the distillation trick frontier open models use to merge RL experts — pcuenq · 2026-11-04
- Stanford preprint: CD19-CAR T cells clear EBV+ autoreactive B cells in autoimmune disease — ahandvanish · 2026-11-04
- Paperclip Datasets: build curated datasets from millions of papers in minutes — james_y_zou · 2026-11-04
- Multi-agent messaging barely helps: 61% of agreed runs still hit merge conflicts — EliasEskin · 2026-11-04
- "Deep lifts" paper: ReLU networks adaptively generalize even with the dumbest random optimizer — yuxiangw_cs · 2026-11-04