Xiaomi Previews MiMo-V3's HySparse2 Architecture, Cutting Million-Token Prefill Compute ~5x
量子位 · wechat · 2026-09-24
Right after MiMo-V2.6 shipped, Xiaomi's Luo Fuli previewed MiMo-V3's new core architecture, HySparse2, built for the agent workload where tool calls emit dozens of tokens but return tens of thousands:
- Early prefill exit: the model splits into a front self-decoder (full + sliding-window attention) and a rear cross-decoder (full + sparse attention), bridged by KV Bridging on full-attention layers only. Prefill stops after the front half; with prefill/decode disaggregation, a 49-layer model needs only the first 25 layers on prefill nodes, nearly halving serving memory.
- Token-level sparse selection: replacing block-based (64-token) selection with per-token selection (1024 global tokens + 128 forced recent tokens) gains +6.57 on RULER-v2 and +8.14 on two-needle MRCR-v2 in ablations.
- Two-level KV sharing: sparse layers reuse full-attention KVCache and token selections; the standalone sliding-window branch is dropped.
Results at 80B total / 3B active params: at 1M context, prefill compute is 1/5.02 of HybridSWA, and FP8 KVCache drops from 12.09GB to 2.69GB (4.5x). At 256k context, RULER-v2 hits 58.45 vs 32.61 (HySparse) and 35.74 (HybridSWA); average MRCR-v2/RULER-v2 gains over HySparse are +11.30 and +19.81 points, with general knowledge/code roughly on par. Evaluations extend to 256k; real-world latency awaits MiMo-V3's release.
More from Models
- HackerGLM games the grader: guesses 11 hidden test IDs to pass a buggy RL environment — echen · 2026-11-12
- ADM 3 Cloud flops in image evals: ranks #86 of 107 in text-to-image — ArtificialAnlys · 2026-11-12
- YC-backed Monte launches post-training platform to build proprietary models from your agents' real work — ycombinator · 2026-11-12
- Leak Claims OpenAI Will Launch Entire GPT-6.2 Family Tomorrow to Counter Opus 5.5 and Fable 5.2 — heypearlai · 2026-11-12
- ARC-AGI-3 leaderboard: models score 90%+ but at $10K+ per task — haider1 · 2026-11-12
- LMArena Launches Agent Arena for Head-to-Head Agent Evaluation — arena · 2026-11-12