Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation
Yeonkyeong Lee, Hyunsung Go, Jongmin Kim, Sewoong Lim, Donghoon Lee
cs.CV
2026-08-25
KATok learns a keep-or-drop gate over video latents, using 366 tokens on 256²×16 with 31.24 PSNR, and trains generation about 6.9× faster than OmniTokenizer.
Latent diffusion is the default path for video: a VAE compresses the clip, then diffusion or flow matching runs in that latent. Conventional VAEs use a fixed ratio, so token count grows linearly with resolution and length. Large regions of a video are static background or flat texture. A fixed budget still spends compute on those empty cells.
Variable-length tokenizers such as FlexTok and ElasticTok let the user, or an inference-time search, pick how many tokens to keep. Flexibility is not adaptivity. KATok wants the model itself to read content complexity, drop uninformative tokens, and spare the generator a budget search.
The encoder is a transformer with 3D RoPE. Clips are cut into 16²×8 patches; each token parameterizes a diagonal Gaussian. A light importance head emits keep/drop logits. Training uses Gumbel-Softmax soft masks multiplied onto the latents; inference uses hard cuts. The same soft mask shifts decoder attention logits, so dropped tokens are invisible on the decode side as well.
An ℓ1 penalty on the soft masks is annealed: roughly the first 5k steps keep almost everything, then sparsity ramps up. The decoder is a FLUX-style dual stream with learnable queries. Encoding stays coarse while decoding uses a finer 8²×4 grid by adding queries, not extra latents. Two register tokens act as global anchors.
Sparse latents misalign under generation: surviving tokens no longer sit on their original spatio-temporal slots. Two fixes. Joint generation flow-matches content and position with decoupled noise schedules. Cascaded generation first predicts an active-cell mask with an 8.3M mask prior, then feeds those positions into the main SiT-XL flow model. Cascaded is the default.
About 5,000 Panda-70M validation clips:
| Method | Resolution | Mean tokens | PSNR | rFVD |
| Omni-VAE | 256²×16 | 5120 | 28.10 | 7.84 |
| ElasticTok-KL | 256²×16 | 3846 | 30.52 | 12.37 |
| KATok | 256²×16 | 366 | 31.24 | 5.12 |
| Omni-VAE | 512²×32 | 36864 | 24.07 | 16.85 |
| KATok | 512²×32 | 1554 | 33.23 | 6.40 |
Compression ratio rises from 134 to 253. Token count tracks temporal entropy (Pearson r about 0.87) more than spatial entropy (0.62). A solid-white 256²×16 clip uses 28 tokens.
Generation gFVD at 100K iterations, lower is better:
| Method | Sky | UCF-101 | Kinetics-600 |
| Omni-VAE | 23.28 | 100.00 | 206.58 |
| ElasticTok | 95.53 | 712.56 | - |
| KATok-Cascaded | 23.19 | 61.53 | 160.84 |
| KATok-Joint | 21.36 | 73.16 | 193.78 |
At 200k UCF-101 steps, Cascaded reaches 49.34 gFVD against OmniTokenizer's 82.31. At 80k it already hits 73.81, beating OmniTokenizer's 200k number, about 6.9× wall-clock on 8 H200 GPUs. Throughput is 15.71 videos/sec, 3.2× OmniTokenizer. Removing the soft attention mask or Gumbel-Softmax collapses training to the two register tokens.
The tokenizer bottleneck is often allocation, not raw compression. Adaptive dropping here both cuts tokens and improves generation. Token count even acts as a motion knob: 200 tokens yield simpler clips, 400 tokens more motion. For long or high-resolution latent video, this is a cheaper, faster-converging latent.
The main tables stay inside transformer plus continuous KL VAEs. CNN VAEs such as Cosmos and LTX-Video are appendix comparisons.
No standalone limitations section. Generation is 256²×16 unconditional or class-conditional, not large-scale text-to-video. ElasticTok was skipped on Kinetics for compute. Cascaded generation needs an extra mask prior; joint generation is sensitive to the two noise schedules. Latent regularization slightly hurts reconstruction to help generation, a deliberate trade. Token count as a control signal is an emergent correlation, not a guaranteed conditioner. Affiliation is Kakao; the project page is kakao.github.io/KATok.