Nanbeige Open-Sources Depth Hybrid Attention, Beating Full Attention

Nanbeige has open-sourced Depth Hybrid Attention (DHA), which mixes values across depth in hybrid attention architectures, allowing layers to leverage cross-layer information and outperform full attention.

2026-11-03 ~ 2026-11-03 · 2 related posts

• Nanbeige's Depth Hybrid Attention beats full attention with tiny extra cost — teortaxesTex · 2026-11-03

1 near-duplicate retellings: syedshad