StepFun open-sources Step 5 Preview: 600B-param MoE, 27B active, 1M context
teortaxesTex · x · 2026-10-15
StepFun has open-sourced Step 5 Preview weights: a 600B-total-param sparse MoE with 27B active per token, native text+vision, 1M-token context, aimed at long-horizon agent tasks, coding and finance. Weights are on Hugging Face (incl. BF16), deployable via vLLM/SGLang with OpenAI-compatible API, plus an open-source agent tool Step Code.
teortaxesTex dissects the architecture: 92-layer narrow backbone (4096-d hidden), only 88 MoE layers with 4 dense SwiGLU MLPs (int. dim 13824 vs 1536 for experts), 352 routed experts with top-8 routing, hybrid 512-token sliding-window + global attention, a compressed-block sparse indexer over 8-token regions (top-512), a new sparse-region GQA variant, a 47-layer vision encoder (0.5Mp → 169 tokens), and 3 multi-token prediction layers for speculative decoding.
More from Models
- Ultramarine 1 Flash: scaling RL for non-verifiable knowledge work, +4.1 pts — lmoroney · 2026-10-15
- OpenAI's Tibo says reports of the Codex reset button being lost are 'greatly exaggerated' — BMarinov · 2026-10-15
- Enterprise AI risk indicator up 9.3x in 2 years, SaaSletter says it leads Anthropic's revenue — thedealdirector · 2026-10-15
- Unverified: Anthropic rumored to be training models to play video games — imjustnewatai · 2026-10-15
- Maya-2 debuts as open-weight Indic voice model, ranks #2 on Hindi TTS leaderboard — CurieuxExplorer · 2026-10-15
- Models may detect AI-generated text by fingerprint alone, and eval sandboxes could be compromised — scaling01 · 2026-10-15