Step 5 Preview architecture blog details Sparse GQA with block-wise token merging cutting indexer cost 8x
teortaxesTex · x · 2026-10-15
StepFun's Step 5 Preview architecture blog reveals Sparse GQA, a sparse attention mechanism built for long-horizon agentic tasks and 1M-token contexts.
Key points:
- Full attention becomes the main training/inference bottleneck at 1M tokens; the team introduces an optimized sparse attention to accelerate it.
- Three bottlenecks are addressed: indexer cost, GPU utilization over selected content, and the granularity-vs-data-read trade-off.
- Existing sparse attention like DSA has non-trivial indexer overhead that can exceed the sparse FlashAttention operator itself; instead of sharing one indexer across GQA groups, they adopt block-wise token merging (block size 8), cutting indexer and top-k selection cost by 8x.
- Adjacent tokens' top-k selections overlap heavily, so the implementation groups overlapping computations to improve GPU utilization.
- Final config B=8, k=4096: at long sequences indexing savings outweigh extra main-path cost, with token coverage close to the B=1, k=2048 reference (78.06% vs 79.09%) and 7.89s prefill latency saved at 1M context.
teortaxesTex highlighted the relatively simple design centered on block-wise token merging.
Related event: Stepfun Open-Sources Step 5 Preview with 600B MoE and 1M Context(2 posts)→
More from Models
- kalomaze argues ensembled synthetic data can cover data-scarce domains — kalomaze · 2026-10-15
- Newer image models are easier to jailbreak, with 2x more attacks getting through — Scobleizer · 2026-10-15
- Gemini controls a real robot from one prompt: SSH in, find joints, learn from demo — heiga_zen · 2026-10-15
- Tokenization refactor speeds up Sentence Transformers, biggest gains for static embedding models — tomaarsen · 2026-10-15
- Hugging Face tokenizers: another ~5x speedup for BERT tokenizers on top of v1 — tomaarsen · 2026-10-15
- Open-weights TYGER model combines ASR with full-duplex turn control, 250ms latency — KokaOP · 2026-10-15