PWT Attention: Training-free 2.5x Speedup for Video Generation
青稞AI · wechat · 2026-07-31
This article delves into attention mechanism optimization, pointing out the natural complementarity between sparse attention (discarding low weights) and linear attention (global linear approximation).
The author analyzes existing methods like SLA and PISA, introducing an improved approach called PWT (Piecewise-Taylor). By introducing the second-order Taylor expansion term, PWT compensates for PISA's zero-order limitation, using block covariance matrices to correct mass underestimation.
Key Advantages & Details:
- Training-free: PWT's exact and approximate branches share the same denominator, allowing it to replace full attention directly without fine-tuning.
- Performance: At 85% sparsity, it achieves a 2.5x end-to-end training speedup (forward + backward) over full attention, with higher single-step approximation accuracy than PISA.
- Engineering Pitfall: In real DiT activations, a few outlier dimensions cause the second-order term to explode. The author solved the generation collapse by introducing an effectiveness threshold to gate (zero out) extreme values.
The forward/backward kernel implementations for PWT have been open-sourced.
More from Infra
- Microsoft hits $90B Q4 revenue, driven by AI growth — emmanuelvivier · 2026-07-31
- AI Data Center Noise Hits 105 Decibels, Sparking Lawsuits Against Tech Giants — mkheck · 2026-07-31
- Yahoo Enhances Search Retargeting with Amazon Bedrock, Boosting Keyword Expansion by 600x — AWS ML Blog · 2026-07-31
- Analyst: AI Fundamentals Unchanged Amidst Severe Compute Deficit and Low Penetration — BenBajarin · 2026-07-31
- AI Buildout Validates Kleiner Perkins' Cleantech Fund 20 Years Later — matt_slotnick · 2026-07-31
- China's DUV Lithography: Not an ASML Killer, But an Iteration Loop — demian_ai · 2026-07-31