Baidu open-sources ERNIE 5.5 Flash: 280B-param MoE with 14B active and 1M context

teortaxesTex · x · 2026-11-05

Baidu released and open-sourced ERNIE 5.5 Flash, a 280B-total/14B-active MoE with notable architecture choices:

Specs: 43-layer backbone (1 dense FFN + 42 Multi-Router Latent MoE layers), 512 routed experts (top-10) plus 1 shared expert per layer; trained from scratch on 33.5T tokens with the Muon optimizer, 128K training context, 1M context for fine-tuning and inference. FP8 and BF16 safetensors are on Hugging Face and ModelScope.
Hybrid attention: 37 Heavily Compressed Attention (HCA) layers caching one entry per 128 tokens, plus 6 MLA layers with sparse attention (DSA). HCA layers use wide 512-dim heads while MLA layers use narrower 256-dim heads; at equal KV-cache budget, HCA+MLA beats HCA+CSA.
Other tricks: low-rank gated attention cuts gate-projection params by 75%; Virtual Head Attention adds a lightweight low-rank post-mix across heads; Multi-Router Latent MoE replaces a single router with several independently initialized sigmoid routers for more accurate expert selection without extra FLOPs or communication.

Observers joked it's 'another DeepSeek V4 Flash,' nodding to the architectural resemblance.

More from Models