Muse Spark Open architecture: latent MoE with 256 experts per layer, 46 SWA + 16 full layers
vllm_project · x · 2026-10-16
A deep dive into Muse Spark Open's architecture: learned pooler/unpooler projections shrink the 8,192-wide hidden state to 4,096 for expert computation (latent MoE). All 62 decoder layers are MoE with 256 experts each. Attention is hybrid — 46 layers use a 2,048-token sliding window while 16 attend to full context. Image/video inputs go through a Perception Encoder, with a separate 118M audio encoder.
Related event: Meta Open-Sources Muse Spark Open, an 816B-Parameter MoE Model(44 posts)→
More from Models
- Underdog Ships Local Image Generation, Stripe Agent Payments; Saluki 27B Hits 300K Downloads in Week One — heyneighbor · 2026-10-16
- Perplexity Computer Ships GPT-6 Astra for Max, Open-Sources SPACE Sandbox Under Apache 2.0 — heyneighbor · 2026-10-16
- Microsoft trained MAI-Base-2 on 16k GB200 GPUs in just 38 days, with 21% MFU — scaling01 · 2026-10-16
- Microsoft's MAI-Base-2: 2.8T-param MoE pretrained on 16k GB200s in just 38 days — scaling01 · 2026-10-16
- Reportedly open-weights Muse Spark Open releases, runs locally on Mac Studio (unverified) — awnihannun · 2026-10-16
- Critics mock Gemini 4 Argon for failing at basic Gmail and Calendar tasks — firstadopter · 2026-10-16