VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
Xingyang Li, Dongyun Zou, Shining Zhang, Jiacheng Chen, Haocheng Xi, Lvmin Zhang, Jun-Yan Zhu, Song Han, Zhekai Zhang, Yujun Lin, Muyang Li
cs.CV
2026-09-15
Training-free VC-Attention smooths values with online k-means and writes E4M3 softmax with one FMA, running 1.59× faster than BF16 FlashAttention-4 on B200.
Video Diffusion Transformers flatten a latent clip into one long spatiotemporal sequence, so attention dominates serving cost. A 5-second 720p Wan2.2-14B clip is about 70K tokens; on an RTX 5090, attention is more than 64% of wall time. Low-bit Tensor Cores can double (FP8) or octuple (FP4 on 5090) the two matrix products, QK and PV. Two steps in between do not keep up.
Accuracy first. A block's quantization scale is set by its largest entries, so a few outliers starve every other value of bits. SageAttention and friends already smooth query and key channel outliers. Value outliers sit on a few tokens, and the channels they spike in shift across heads, layers, and denoising steps. A rotation preserves a token's norm; a static layout rarely groups the right neighbors. On Wan2.2, after QK error is reduced, the value term is still 82% of the output error.
Speed next. B200 doubled Hopper's Tensor Core throughput and left the exponential unit where it was. Once both products run in FP8, the FP32 exponential and the FP32-to-E4M3 cast become the longest pipeline stage. Attn-QAT reports at most 1.3× over BF16 FlashAttention-4 on B200, and a slowdown if values are quantized on the fly.
VC-Attention is training-free. It patches two steps inside a FlashAttention-4 kernel.
V-Smooth restores accuracy. An online k-means over value tokens, per batch and head, yields a permutation applied to keys and values together. For non-causal self-attention that permutation leaves the output unchanged. Each hardware block of 128 rows subtracts its mean; only the residual is quantized, per-channel E4M3 at 8 bits and NVFP4 at 4 bits. The mean is a 16-bit vector, 0.125 bit per element, and is restored as an outer product from the row sum that online softmax already keeps. No second pass over V.
Clustering is the point of the permutation. In sequence order the block mean removes 8% of block energy; DeltaQuant's fixed spatiotemporal cube removes 12%; sorting removes 36%. A Hadamard rotation of V changes error by 0.2%. Grouping adds 30% to attention time on a step that groups, so the deployed schedule groups only the first 25% of denoising steps and reuses the permutation across neighbors. Averaged over the trajectory, grouping is 3% to 4% of attention time.
ExpCast-FP8 restores speed. An E4M3 byte is already an affine function of log2 of the value it stores, so the kernel writes that byte from the log-domain score with one fused multiply-add, round-to-nearest-even, and a clip into the normal range. The constant β = -0.35 centers the leftover mantissa error; it is read off the format, not fitted per model. For rows that stay in range, total variation is bounded by 3.64%. On 204.8K Wan2.2 rows the mean is 1.6%.
Preprocessing (RoPE, means, gather, Hadamard, quantizers) is fused into the quantizer so the permuted high-precision tensors never hit HBM. The handwritten CuTe/CUDA chain is 8.74× faster than the unfused passes. Datacenter GPUs run both mechanisms at 8 bits. Workstation Blackwell runs V-Smooth only at 4 bits: an on-the-fly NVFP4 probability tile puts its per-16-element scale back on the softmax critical path.
Every fidelity number is scored against the BF16 FlashAttention-4 output of the same model and seed, 100 MovieGen Bench prompts.
| Method | Wan2.2 PSNR | HunyuanVideo-1.5 PSNR | Attention vs BF16 |
| SageAttention2 8-bit | 20.3 | 15.6 | slower than BF16 on B200 |
| V-Smooth 8-bit | 22.6 | 18.4 | accuracy config |
| V-Smooth + ExpCast | 20.5 | 17.5 | 1.59× B200 / 1.46× H200 |
| SageAttention3 4-bit | 13.9 | 10.9 | similar cost to 4-bit V-Smooth |
| V-Smooth 4-bit | 16.8 | 13.8 | 2.27× PRO 6000 / 3.58× 5090 |
At 8 bits, V-Smooth beats SageAttention2 by 2.3 dB on Wan2.2 and 2.8 dB on HunyuanVideo-1.5, and cuts LPIPS 13% to 29%. Fusing ExpCast gives back 0.7 to 2.1 dB and still leads SageAttention2 on all four models. At 4 bits, V-Smooth beats SageAttention3 by 2.9 dB on Wan2.2 and 3.6 dB on LongCat-Video. Adding a QK Hadamard to the baselines moves PSNR by at most 0.1 to 0.3 dB. The leftover error is not in QK.
End to end on one Wan2.2 clip: 1.19× on B200, 1.13× on H200, 1.36× on the RTX PRO 6000, 1.70× on the RTX 5090. On B300 with both products in FP8, attention is 1.47× at 18.4 dB, against 1.31× and 17.1 dB for a naive FP8 kernel.
Run training-free, Attn-QAT sits 3.4 to 6.7 dB below SageAttention2 and is the only method that perturbs VBench. Its published 4-bit numbers need retraining.
For video DiT serving, quantized attention is no longer a QK-smoothing problem. Token-level value outliers and the scalar softmax path are the bottlenecks on B200-class GPUs. VC-Attention leaves the attention pattern and the sampling schedule untouched, so it stacks with sparse attention, feature caching, and few-step distillation. On workstation cards the 4-bit clip speedups of 1.36× to 1.70× are the number that matters for local generation.
This is a kernel-level increment, not a new generative recipe. The ceiling is attention's share of the forward pass: 1.59× on the kernel becomes 1.19× on the clip.
V-Smooth assumes non-causal self-attention: permuting K with V is what keeps the output identical, so a causal decoder cannot copy the trick. Grouping covers only the first quarter of denoising steps; spreading the same number of grouping steps uniformly loses 2.6 dB, so the permutation is less reusable later in the schedule. ExpCast trades accuracy for speed. On MiniMax-H3 the fused kernel's 20.2 dB matches 8-bit FlashAttention-4 and sits 0.3 dB above SageAttention2's 19.9, inside the 0.6 dB band of the strongest training-free baselines. There is no 4-bit datacenter configuration. Fidelity is PSNR/SSIM/LPIPS against BF16; VBench subject consistency and imaging quality are saturated and do not separate methods. SageAttention2 runs at 0.29× of BF16 on B200 because it ships no Blackwell kernel.