Qwen vs MiniMax Sparse Attention Compared as New TileLang Kernels Drop
On August 28, technical author stochasticchasm published a series of observations and results on Qwen's sparse attention mechanism, sparking discussion about training strategies and reasoning-capability gains.
Confirmed
- The author argues Qwen's sparse attention is a refined version of MiniMax's: both are block sparse, with the key difference being pooling position — Qwen pools before SDPA (scaled dot-product attention), while MiniMax pools after (m1/m4 restate the same point).
- The author disagrees with Qwen's two-stage "freeze the full network → distill the indexer → unfreeze" training recipe, and would like to see from-scratch training and sparsification during CPT tested as comparisons, arguing MiniMax does this better (m2, m3).
- The author released another TileLang kernel library with benchmark comparison charts: gains on reasoning-type (non-knowledge-dependent) benchmarks are significantly larger than on other types (m5, m6).
Why It Matters
- The difference in pooling position relative to SDPA is a key lens for understanding the two designs' trade-offs, potentially affecting long-context efficiency-vs-accuracy balance.
- Whether two-stage training is necessary affects training cost and reproducibility; if from-scratch training or CPT-stage sparsification performs comparably, it would simplify the training path for sparse-attention models.
- Reasoning benchmarks improving far more than knowledge ones suggests the new kernel or model's gains concentrate in pure reasoning ability — useful for benchmark selection and capability attribution.
2026-08-28 ~ 2026-08-28 · 6 related posts
Primary sources
- Analysis compares Qwen and MiniMax sparse attention implementations — stochasticchasm ·
- Critique of Two-Stage Training and Comparison with Minimax — stochasticchasm ·
- New TileLang Kernel Library Shows Major Gains — stochasticchasm ·
- Reasoning benchmarks show significantly higher uplift than others — stochasticchasm · 2026-08-28
- [source] New TileLang Kernel Library Shows Major Gains — stochasticchasm · 2026-08-28
- [source] Analysis compares Qwen and MiniMax sparse attention implementations — stochasticchasm · 2026-08-28
- [source] Critique of Two-Stage Training and Comparison with Minimax — stochasticchasm · 2026-08-28
- Discussing Training Strategy in Dense Distillation Phase — stochasticchasm · 2026-08-28
1 near-duplicate retellings: stochasticchasm