Xiaomi MiMo next-gen moves to causal encoder-decoder: 5x less prefill compute at 1M context
joey00072fp8 · x · 2026-11-08
Per SemiAnalysis, following DeepSeek V4.1 Flash, Xiaomi's MiMo is moving its next-gen models to a causal encoder-decoder architecture dubbed HySparse2.
Key design: prompts pass through only 25 of 49 layers (a causal encoder with SWA×12 + full attention + SWA×12). Decoder KV caches are projected from encoder hidden states via KV Bridging, with four cross-decoder full-attention layers each holding distinct K/V projections, each surrounded by 5 sparse attention layers that reuse that block's KV and selection indices (1,024 global + 128 recent tokens per block).
At 1M context vs MiMo-V2: 5x less prefill compute and a 4.5x smaller KV cache. The accompanying diagram shows the attention and cache-reuse scheme in an 80B-A3B research configuration.
More from Models
- $50 OpenAI Pro tier rumored; Pro plans offer 10x–20x Plus usage with no 5-hour caps — Ok_Homework_1859 · 2026-11-08
- Gemini 4 'Carbon' rumored to launch next week, public rollout possibly Nov 10 — haider1 · 2026-11-08
- Claude's new checkpoint leaks that Karpathy joined Anthropic's pretraining team in May 2026 — ChrisGPT · 2026-11-08
- Shruti open-weights speech-to-text covers all 22 Indian languages offline in under 30MB — kingslyj · 2026-11-08
- Follow-up: the new Claude checkpoint performs 'a bit too good' for a Sonnet — ChrisGPT · 2026-11-08
- OpenAI AI-generated manuscript claims proof of local smoothing conjecture in all dimensions — jiqizhixin · 2026-11-08