Xiaomi Previews MiMo-V3's HySparse2 Architecture, Cutting Million-Token Prefill Compute ~5x
量子位 · wechat · 2026-09-24
Right after MiMo-V2.6 shipped, Xiaomi's Luo Fuli previewed MiMo-V3's new core architecture, HySparse2, built for the agent workload where tool calls emit dozens of tokens but return tens of thousands:
- Early prefill exit: the model splits into a front self-decoder (full + sliding-window attention) and a rear cross-decoder (full + sparse attention), bridged by KV Bridging on full-attention layers only. Prefill stops after the front half; with prefill/decode disaggregation, a 49-layer model needs only the first 25 layers on prefill nodes, nearly halving serving memory.
- Token-level sparse selection: replacing block-based (64-token) selection with per-token selection (1024 global tokens + 128 forced recent tokens) gains +6.57 on RULER-v2 and +8.14 on two-needle MRCR-v2 in ablations.
- Two-level KV sharing: sparse layers reuse full-attention KVCache and token selections; the standalone sliding-window branch is dropped.
Results at 80B total / 3B active params: at 1M context, prefill compute is 1/5.02 of HybridSWA, and FP8 KVCache drops from 12.09GB to 2.69GB (4.5x). At 256k context, RULER-v2 hits 58.45 vs 32.61 (HySparse) and 35.74 (HybridSWA); average MRCR-v2/RULER-v2 gains over HySparse are +11.30 and +19.81 points, with general knowledge/code roughly on par. Evaluations extend to 256k; real-world latency awaits MiMo-V3's release.
Related event: Xiaomi Reveals HySparse2 Architecture for MiMo-V3(5 posts)→
More from Models
- Community asks why BFL video model and Minimax H3 weights never shipped despite open-source promises — krigeta1 · 2026-09-24
- Putting a Decision Model Like JEV in Production: Thresholds Are Margins, Not Accuracy — sven_ai · 2026-09-24
- Users find Opus 5.5 has remarkably strong hearing, works via spectrograms — repligate · 2026-09-24
- GPT-6 Astra agent plays Left 4 Dead 2 in first real-time run — imjustnewatai · 2026-09-24
- Same Prompt, Opposite Results: GPT-4 Goes Silent 30/30 Where GPT-3.5 Never Stops — rayanpal_ · 2026-09-24
- Box says Opus 5.5 cuts token usage 63% and runs 30% faster than Opus 5 — bcherny · 2026-09-24