Baidu open-sources ERNIE-5.5-Flash-Base: 280B MoE activating just 14B per token
智东西 · wechat · 2026-11-05
Baidu released and open-sourced ERNIE-5.5-Flash-Base, a sparse MoE base model with 280B total parameters activating only 14B per token, pretrained from scratch on 33.5T tokens with 128K training context and 1M-token inference support. Baidu claims same-size SOTA, leading rivals like DeepSeek-V4-Flash-Base and MiMo-V2.5-Base on reasoning, code, and long-context tasks.
Three core architectural changes:
- Hybrid sparse attention: 37 of 43 layers use heavy-compression attention (HCA) compressing 128 tokens per cache entry; 6 layers retain MLA with dynamic sparse attention (DSA) for precise retrieval, slashing KV cache costs.
- Multi-Router Latent MoE: multiple independent sigmoid routers jointly score experts for more robust routing at zero extra compute; experts compute in a 2048-dim latent space, roughly halving FLOPs and dispatch/combine traffic.
- ECHO future-token prediction: reuses the trunk's last layer instead of a separate MTP module, doubling as a built-in speculative-decoding draft model with higher acceptance rates.
Weights are on Hugging Face (FP8/BF16), with vLLM and SGLang ecosystem support.
More from Infra
- 40 of 41 Data Center Ballot Measures in 2026 Midterms Went Against AI Builders — Acceptable-Offer-518 · 2026-11-12
- Azure Day at KubeCon: AKS Runs 22,655 Agents on 65 Nodes in Live Demo — davemccollough · 2026-11-12
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12