Qwen's sparse attention compared to MiniMax; TileLang kernel boosts reasoning benchmarks

On August 28, technical author stochasticchasm posted a series of observations on Qwen's sparse attention and released a TileLang kernel library with benchmark comparisons, sparking discussion about sparse attention design trade-offs and training strategy. The takeaway: Qwen's approach looks like a refined version of MiniMax's sparse attention—both block-sparse, differing mainly in pooling placement—and benchmarks show the largest gains on reasoning (non-knowledge) tasks.

Confirmed

• Both Qwen and MiniMax use block-sparse attention; the key difference is that Qwen pools before SDPA while MiniMax pools after.
• stochasticchasm disagrees with Qwen's two-stage pipeline (freeze network → distill indexer → unfreeze), calling for comparisons against training from scratch or sparsifying during CPT, which he considers MiniMax does better. In a separate post he also discusses how using teacher forcing from the start of the dense distillation phase might perform instead.
• He released a TileLang kernel library with cross-benchmark comparisons: gains vary widely, with reasoning (non-knowledge) benchmarks improving the most.
• Developer codestar, citing MultiPL-E results, noted large variance differences across benchmarks worth investigating, and made the same observation that reasoning benchmarks gained the most.

Why it matters

• Pooling placement relative to SDPA is central to understanding the two designs' trade-offs in efficiency and accuracy for long-context processing.
• Whether two-stage training is necessary affects training cost and reproducibility; equivalent results from scratch or CPT-stage sparsification would simplify the pipeline.
• Gains concentrated in reasoning benchmarks suggest the benefit lies mainly in pure reasoning ability, informing benchmark selection and capability attribution; the benchmark-variance question raised by codestar also cautions against over-interpreting eval results.

2026-08-28 ~ 2026-08-28 · 7 related posts

• Reasoning benchmarks show significantly higher uplift than others — stochasticchasm · 2026-08-28
• [source] New TileLang Kernel Library Shows Major Gains — stochasticchasm · 2026-08-28
• [source] Analysis compares Qwen and MiniMax sparse attention implementations — stochasticchasm · 2026-08-28
• [source] Critique of Two-Stage Training and Comparison with Minimax — stochasticchasm · 2026-08-28
• Discussing Training Strategy in Dense Distillation Phase — stochasticchasm · 2026-08-28
• Benchmarks vary wildly, and reasoning tests get the biggest uplift — code_star · 2026-08-28

1 near-duplicate retellings: stochasticchasm