Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models
Pardis Taghavi, Reza Langari, Gaurav Pandey
cs.CV, cs.AI, cs.LG
2026-08-19
SparsePR fits residuals from 64 exact rows after response-coupled partitioning. Four video DiTs match dense quality at 22-26% pair density, with 1.48-2.61x end-to-end speedups.
Video DiTs and world models run long spatiotemporal sequences, so quadratic attention dominates high-resolution, long-duration inference. Training-free block-sparse attention already exploits spatiotemporal structure, picks important blocks online, permutes tokens into executable layouts, or approximates skipped interactions. Row-wise concentration does not specify the support a group of queries needs when they share one block route.
On Wan2.2, a median 6.2% of keys keeps 90% of a single query's attention mass; eight queries sharing one route push the union to 22.9%. Cosmos-Predict2.5 is already dense per row (median 56.5%), and pooling lifts that to 77.7%. HunyuanVideo stays sparse at both scales. Shared-route density can come from poor support overlap inside a group, or from rows that were never sparse.
Under renormalized hard drop, the residual equals omitted mass times the gap between the omitted-support output and the sparse output. Similar retained mass does not bound that gap. Cosmos3-Nano is sensitive to this in sparse runs. Partition geometry also changes how much of the leftover residual is an affine function of the sparse output. An executable sparse operator has to handle both shared-route support and the residual from skipped pairs.
SparsePR, from Texas A&M, is training-free and runs entirely inside the current attention call.
Response-Coupled Partitioning samples queries from that call and groups keys by pre-softmax response. Keys with similar response profiles share a K/V block; values follow their paired keys. Centroids of those key groups then define query coordinates, so queries with similar response direction share a route. One asymmetric pass, no alternating query/key refinement. Feature ranks are rK=48 and rQ=64. Nearby activations do not imply similar routing; grouping in current-call response geometry keeps pooled support tighter and makes the residual more affine in the sparse output. Versus a semantic k-means partition, the affine-explainable residual fraction rises 3.2 to 14.9 points, and affine-orthogonal energy falls to 0.285x-0.653x.
Probe-Fitted Residual Reconstruction evaluates M=64 query rows exactly against all keys, measures R = Odense - Osparse, and fits a ridge affine map (lambda=0.1) from the sparse output to that residual. The feature-dependent correction is then projected onto the r=16 output subspace seen in the probe residuals. Probes are stratified across query groups and ordered by distance to the group centroid. Sixty-four rows are 0.054%-0.145% of pairs at the sequence lengths tested. The method does not claim the full residual matrix is low rank; it only refuses to extrapolate into unseen output directions.
Q, K, and V are permuted into group-major order. Selected cells go through FlashInfer's variable-block sparse kernel and are renormalized over retained K/V groups. Under grouped-query attention, K/V partitions and routes are shared across associated query heads. Total executed-pair density is routing density plus M/Nq; the router reserves probe cost first. Targets are 22% for HunyuanVideo, Wan2.2, and Cosmos-Predict2.5, and 26% for Cosmos3-Nano. Realized density can drift slightly because whole cells are atomic. Timing includes partitioning, routing, permutation, the sparse kernel, probes, the fit, and write-back.
All runs use BF16 on one NVIDIA H100 at 720p, with matched conditioning, seeds, steps, guidance, resolution, and frame count.
| Model | Method | PSNR↑ | ImgQual↑ | SubCons↑ | Density | Speedup |
| HunyuanVideo-13B | Dense | - | 0.850 | 0.976 | 100% | 1.00x |
| HunyuanVideo-13B | SVG2† | 30.452 | 0.852 | 0.927 | 25.45% | 2.30x |
| HunyuanVideo-13B | SVG-EAR† | 31.043 | 0.845 | 0.903 | 22.17% | 1.93x |
| HunyuanVideo-13B | SparsePR | 31.844 | 0.850 | 0.976 | 21.92% | 2.61x |
| Wan2.2-I2V-A14B | Dense | - | 0.689 | 0.974 | 100% | 1.00x |
| Wan2.2-I2V-A14B | SparsePR | 30.658 | 0.687 | 0.973 | 21.97% | 1.80x |
| Cosmos-Predict2.5 | Dense | - | 0.714 | 0.976 | 100% | 1.00x |
| Cosmos-Predict2.5 | SparsePR | 26.328 | 0.714 | 0.976 | 22.14% | 1.51x |
| Cosmos3-Nano | Dense | - | 0.700 | 0.950 | 100% | 1.00x |
| Cosmos3-Nano | SparsePR | 24.417 | 0.699 | 0.949 | 25.96% | 1.48x |
On HunyuanVideo, image quality and subject consistency match dense exactly, PSNR 31.844 beats SVG-EAR's 31.043, compute drops from 612.38 PFLOPs to 255.95, and end-to-end speedup is 2.61x. On Wan2.2, dense scores 0.689 / 0.974 on image quality and subject consistency; SparsePR scores 0.687 / 0.973 with LPIPS 0.044 versus SVG-EAR's 0.093. Full generation falls from 1650 s to 917 s. World models accelerate less: SVG2 is 1.24x and SVOO 1.03x on Cosmos-Predict2.5, SparsePR 1.51x, with PBench 77.75 against dense 77.76. Cosmos3-Nano needs 25.96% density to hold quality, PBench 77.30 versus 77.31, speedup 1.48x.
Ablations separate the two parts. Hard drop only, Hunyuan mean/p99 error moves from 0.0887 / 0.7136 (semantic) to 0.0736 / 0.6967 (response-coupled). After probe repair, semantic partitioning reaches 0.0527 / 0.3562 and full SparsePR 0.0330 / 0.2285. On Cosmos-Predict2.5, hard-drop mean error sits near 0.76-0.79; probe repair cuts semantic partitioning to 0.2622 and SparsePR to 0.0954. Probe fitting removes most of the error; response-coupled partitions make a finite probe set more useful. Stratified probes beat random and uniform spatiotemporal sampling; projecting onto the probe-residual subspace helps a little over ridge alone.
This is inference acceleration for already-trained video DiTs and world models, with no retraining and no distillation. Relative to SVG2 and SVG-EAR, the deltas are concrete: Hunyuan goes to 2.61x versus SVG2's 2.30x, and subject consistency returns from 0.927 to the dense 0.976. On world models SVOO barely moves (1.02x-1.03x); SparsePR still reaches 1.48x-1.51x.
The accounting is honest. Executed-pair density includes probe rows. Latency includes online k-means, permutation, and the fit. Probe fitting is 1.1% of Wan2.2 generation time, so the fidelity gain is cheap.
Models that are dense per row have little room. Cosmos-Predict2.5 needs a median 56.5% support already, and speedup is only 1.51x. This is a systems-level incremental improvement, not a new generation recipe.
The paper has no standalone limitations section. The affine residual is an empirical local model for the active call. Omitted mass and the omitted-support output are not uniquely determined by the sparse output. Corrections live in the output directions the probes saw; unseen directions stay uncorrected.
Speedup differs by more than 1x across models: 2.61x on Hunyuan, 1.48x on Cosmos3-Nano. Cosmos3-Nano still posts LPIPS 0.176 against Hunyuan's 0.087. Matching dense on world models is mostly VBench / PBench scores; frame-level fidelity is weaker.
SpargeAttn, SVG2, SVOO, and SVG-EAR numbers on Hunyuan and Wan carry a dagger mark and come from prior papers; the two Cosmos models were reproduced by the authors under matched hardware and timing. SVOO's Wan image quality of 0.7337 exceeds dense 0.689, which looks like a protocol mismatch, not a fair win. Experiments stay on one H100, 720p, and FlashInfer's variable-block kernel. Longer videos, multi-GPU, and text transformers are untested.