FastFrames: Video Generation at Interactive Latency
Gene Chou, Guandao Yang, Fujun Luan, Lars Mescheder, Wei Dong, Stephan Richter, Vladlen Koltun
cs.CV
2026-11-18
FastFrames replaces VAE decoding and the KV-cache refresh with light modules. At 1.3B it answers in 36 ms on one B200 and matches a baseline 7.6× slower on VBench.
Autoregressive video diffusion extends a clip when a new control arrives. Camera steering, physics, games, and robot prediction all need that interface. Interaction fails on latency. HCI work puts noticeable lag at about 50 to 60 ms, so the budget is 50 ms from a new input to the first RGB frame that shows its effect, cache update and pixel decode included. A model that emits 30 frames in one batched second can report 30 FPS while a person still waits out that second. Responses per second (RPS) is one over that wait. 50 ms is 20 RPS.
Causal Forcing++ (CF++, Wan 2.1 at 1.3B) was the fastest few-step model in the profile. It takes 272 ms on one B200, about 3.68 RPS, roughly 5.4× short of 20 RPS. Two denoising steps are only about 30% of that wall time. VAE decode, the step that turns latents back into pixels, takes 147 ms on its own, about 3.5 passes of the DiT denoiser and about 3× the whole budget. Refreshing the KV cache, the stored keys and values of earlier frames, is another full DiT pass at 41.6 ms, and that pass's prediction is discarded.
FastFrames is Apple's family on the Wan 2.1 backbone: two sampling steps, one latent frame per autoregressive step. Cutting to one step was already measured on the baseline. Quality falls off, and latency only drops about 15%. The time comes out of decode, cache refresh, and the system work around each DiT call.
The Wan VAE decoder has 73.3M parameters and still takes 147 ms, because it runs dense 3D convolutions at every resolution. Four frames at 480×832 means convolution over about 1.6 million spatiotemporal sites. A linear readout (4.7M parameters, 0.06 ms) makes the scene recognizable and smears hair and rain. The replacement is three causal 3D ConvNeXt blocks on the DiT's 21×30×52 grid, then a linear layer that unpacks each token into a 4×16×16 RGB patch. At 63.9M parameters it takes about 1 ms, about 147× faster, and those thin structures stay. The DiT stays frozen. L1, MSE, spatial gradients, frame-to-frame change, and a motion-weighted term match pixels from the frozen VAE. On an 81-frame 832×480 clip in BF16 with activation checkpointing, a VAE-decoder update peaks near 68 GB; this head peaks near 5 GB, which is small enough for an adversarial loss in pixel space. The discriminator is a frozen DINOv3 ViT-S/16, with a spatial head and a temporal head on layers 4, 8, and 12, trained on an internal set of 10K real videos. Finetuning the DiT at a low learning rate was unstable, so the trunk stays frozen. The head also has to be trained once per model. It does not transfer from 1.3B to 14B the way the Wan VAE does.
In a two-step sampler the baseline spends one extra DiT pass writing the clean latent's keys and values into the cache. That is one third of the DiT evaluations, and the output is thrown away. Dropping the pass, or training the model to attend to noisy cache entries, leaves visible artifacts. A small network takes over: the frozen patch embedding, four transformer blocks trained from scratch, and a per-layer residual on K and V. Supervision is the original DiT cache write, plus a loss on the attention output. The network takes about 2 ms at 1.3B and 6 ms at 14B, about 20× and 12× faster than the baseline refresh.
Systems work then cuts one 1.3B DiT pass from 41.6 ms to 16.4 ms on one B200 with a full cache of 32,760 tokens: persistent KV buffers, precomputed RoPE and conditioning, fused QKV projections, and CUDA graphs. One frame does not offer enough queries to fill the GPU, so the KV sequence is split into five segments of 6,552, attention runs in parallel, and a softmax merges the outputs. That is about 13% off. On eight GPUs, only context parallelism paid for its traffic. Queries are partitioned across devices, and each layer exchanges K and V once. DiT time falls to 5.8 ms at 1.3B, and from 75.6 ms to 20.3 ms at 14B.
FastFrames-S (1.3B) starts from the two-step CF++ checkpoint, keeps the DiT frozen, and trains only the pixel head and the KV network. FastFrames-L (14B) is converted from bidirectional Wan 2.1, distilled with distribution matching (DMD) following the CF++ recipe, and then given the same heads.
Every model uses the same VBench prompts and seeds at about 480p. FastFrames runs at 832×480. Autoregressive latency is steady state with a full cache. One response is one autoregressive step. One latent frame unpacks to four RGB frames, so 36 ms is the wait until the next input takes effect.
| Method | Setup | Latency | VBench total |
| CF++ | 1.3B, 1 GPU | 272 ms | 84.11 |
| FastFrames-S | 1.3B, 1 GPU | 36 ms | 84.23 |
| FastFrames-S | 1.3B, 8 GPUs | 14 ms | 84.23 |
| Krea-Realtime | 14B, 1 GPU | 1675 ms | 84.42 |
| FastFrames-L | 14B, 1 GPU | 158 ms | 84.44 |
| FastFrames-L | 14B, 8 GPUs | 45 ms | 84.44 |
S is 7.6× faster than CF++ and 0.12 higher on the total. Quality is 85.30 versus 85.09. Semantic is 79.92 versus 80.20. L reaches 45 ms on eight B200s, about 22 RPS. Against Krea-Realtime at the same 14B size, one GPU is about 10.6× faster. Totals are 84.44 versus 84.42, quality 85.48 versus 85.08, semantic 80.29 versus 81.81. Every other listed model is slower than 50 ms. Bidirectional Wan2.2 scores 84.65 and takes 117,734 ms.
The same replacements transfer to released conditional models. Camera-conditioned LingBot 1.3B drops from 842 ms, the fastest official setup (two GPUs with Ulysses), to 52 ms on eight GPUs, about 16×. Physics-conditioned RealWonder 1.3B drops from 1141 ms on one GPU to 153 ms, and to 51 ms on eight GPUs, about 22×. LingBot 14B drops from 1373 ms to 233 ms on eight GPUs, about 5.9×, still far above 50 ms. Those models take more sampling steps, four for LingBot and three for RealWonder, and each step predicts several latent frames.
After distillation down to two steps, most of the wait is VAE decode plus a full network pass that only refreshes the cache. Both show up across autoregressive video diffusion, so a new backbone or a better distillation recipe can sit on top.
At 1.3B, 36 ms on one B200 is about 28 RPS, under the 50 ms line where people start to notice lag. No baseline in the same table gets there. The 14B model needs eight B200s to reach 45 ms. The speedup is nearly 8×. Quality stays around parity.
The reported evaluation is 480p on B200 GPUs. The paper names the gaps: action conditioning that does not add latency, higher resolution, lighter hardware, and long-horizon consistency together with physical plausibility.
The subscores do not support a flat claim that quality held. S uses the same DiT weights as CF++ and loses 0.28 on semantics. The abstract says the 14B model has significantly better visual quality and realism. Against Krea-Realtime the total is only 0.02 higher, quality is up 0.40, semantics are down 1.52, and the qualitative write-up calls the realism comparable. Adversarial training uses an internal set of 10K videos, and each model needs its own pixel head. A single 14B GPU takes 158 ms. LingBot 14B still takes 233 ms on eight GPUs. A larger trunk or a heavier condition leaves the 50 ms band.