Meta open-sources Muse Spark Open: 816B/42B MoE reasoning model with 1M context, vLLM Day-0 support
vllm_project · x · 2026-10-16
vLLM announced Day-0 support for Meta's Muse Spark Open, an open sparse-MoE reasoning model with 816B total / 42B active parameters and a 1M-token context, built for long-horizon agentic coding with native image and video perception.
- Architecture: all 62 layers are MoE (256 experts, 8 active per token, no shared expert); experts run at 4-bit while router/squeeze matrices stay full precision; 128 query heads share 16 KV heads (8x smaller KV cache); layers alternate sliding-window (2048) and full attention in a 4-layer cycle, with RoPE dropped on long-range layers.
- Serving: NVIDIA and AMD ROCm images available; MXFP4 and BF16 fit on a single 8-GPU node with TP + expert parallelism, plus an optional DSpark draft model for speculative decoding.
- RL: Day-0 LoRA RL with vLLM rollouts and verl FSDP2 training — 150 GRPO steps on 16 GB300 GPUs lifted AIME-2024 mean@8 from 0.56 to 0.85.
More from Infra
- llama.cpp v0.7.0 adds GPU LRU cache for host-memory MoE experts, up to 2.2x faster decode — EAccelerate_42 · 2026-10-16
- NVIDIA ships Nemotron 3.5 Bolt: 74B-A9B model in 24GB of memory — NVIDIAAI · 2026-10-16
- Creative Strategies launches CS Forecast Stack: 19 linked models covering AI's full economic chain — BenBajarin · 2026-10-16
- Copper to optical: optical backplanes seen as necessary as racks move to optics — BenBajarin · 2026-10-16
- Polymarket prices 28% odds a US state enacts data center moratorium by year-end — Polymarket · 2026-10-16
- Pennsylvania residents reject $10,000-per-household offer for massive Amazon data center — Polymarket · 2026-10-16