TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining
Shih-Ying Yeh, Daniel Z. Kaplan, Xuehai Wang, Fu-En Yang, Min-Hung Chen, Shang-Hong Lai
NeurIPS 2026 main track
cs.CV
2026-09-27
TT-VidT pretrains video encoders with a first-frame anchor plus 8 motion tokens per frame, leading Jester, SSv2, ARID and Diving48 at roughly half the encoder FLOPs.
Comparisons in video self-supervised learning tend to evaluate complete recipes: architecture, objective, data exposure, schedule and decoder capacity all shift at once, so a gain cannot be attributed to any single choice. This paper pins the variables: 4 encoders × 6 objectives = 24 cells, all trained under one recipe at 170M–190M parameters on 1.7M clips (OpenVid plus Moments-in-Time v2) for 8 epochs, then probed cell by cell.
The deeper motivation: many action-recognition labels can be predicted from a single frame, using objects, scenes and clothing alone, so a video model's score may reflect memorized appearance, not temporal understanding. The question is which architecture-objective pairs produce motion-prioritized representations, meaning gains concentrated on tasks that depend on frame-to-frame change while staying competitive on appearance-dominated ones.
The encoder has two paths. The wide path is a DINOv3-initialized ViT-B/16 that processes each frame independently and carries appearance. The narrow path is the Temporal Transfer Layer: 8 learnable motion tokens per frame that self-attend with their frame's spatial tokens, then exchange information across frames.
The pretraining objective, Diff Compression, reconstructs frame t from the first frame's full spatial features plus frame t's motion token, with a diffusion or flow-matching loss. It is an information bottleneck by construction: the anchor already holds appearance, so the motion token must carry whatever cannot be recovered from it. VTok, the closest prior, computes its motion token as an explicit feature difference between target and key frame; here the token is learned end-to-end with no built-in subtraction.
The decoder is pretrained separately, and smaller is better: a video-pretrained DiT-S lifts Jester from 53.89 (ImageNet init) to 73.25, while the larger DiT-L drops it to 34.37. The authors' reading is that a stronger decoder absorbs the reconstruction burden itself, so the motion tokens end up carrying less.
Only two of the 24 cells reach the top tier on Jester and SSv2 without augmentation: ViT3D+MAE (standard VideoMAE) and TT3D+Diff Compression. The paired ablations show an interaction effect: TT3D with MAE falls from 53.89 to 11.07 on Jester, and Diff Compression with a ViT3D encoder falls to 19.82. Neither half works alone.
Final comparison, all retrained under the shared recipe with frozen attentive probing:
| Method | HMDB51 | ARID | Jester | SSv2 | D48 FT |
| V-JEPA 2 | 22.70 | 16.01 | 27.41 | 10.59 | 8.02 |
| VideoMAE | 27.73 | 24.37 | 39.47 | 14.95 | 8.43 |
| DisMo (dual aug) | 22.57 | 22.45 | 46.95 | 13.33 | 8.12 |
| TT-VidT | 25.15 | 37.47 | 73.25 | 25.92 | 18.63 |
Relative gains over the strongest non-TT row run 54% to 121%: 26.3 points on Jester, and Diving48 finetuning more than doubles from 8.43 to 18.63. Encoder FLOPs: TT3D at 456.1 GF versus 874.7 for DisMo and 1012.6 for VideoMAE, a 48–55% cut.
The most convincing evidence is the motion-inversion probe. SSv2 clips whose classes are exact mirrors get flipped or time-reversed, re-encoded and re-classified. TT-VidT keeps its original answer on under 1% of clips; every baseline keeps 9–43%, worst under time reversal. The single-frame DINOv3 control keeps 100% under reversal, which validates the probe. After 30 epochs of end-to-end finetuning on Jester, the baselines converge near 51.5, still below TT-VidT's frozen 73.25, while TT-VidT itself barely moves (72.79). The gap survives multi-seed checks and a near-doubled training budget.
The entire study fits in roughly 4,580 GPU-hours (mostly B200), with one encoder pretraining run at 87.7 hours, a reproducible scale. If your downstream task is defined by motion (gestures, temporal order, fine-grained dynamics), this recipe buys 10 to 26 points at half the compute. Three pieces of methodology are portable: the controlled architecture-objective sweep, the per-clip flip probe that tests whether a model actually reads frame order, and the counterintuitive finding that a larger decoder leaves the encoder with weaker motion features. That last one applies to other reconstruction-based pretraining.
The boundary is equally clear: TT-VidT does not lead HMDB51, IARD or EPIC-Kitchens. It is a method with a defined operating domain, not a general-purpose video backbone.
The authors state their own: one data mixture, one 170M–190M scale, most sweep cells run once (multi-seed checks only on canonical configurations), so conclusions cover this controlled budget only. TT3D with DisMo-style dual augmentation, alternative appearance probes and partial unfreezing were not tested.
A few things deserve scrutiny. The uniform no-augmentation constraint structurally handicaps DisMo, whose native scheme depends on motion-preserving augmentation; the paper patches this with a dual-augmentation ablation but leaves TT3D plus dual augmentation untested. The V-JEPA 2 row is an internal small-scale retrain far from its native regime, so those numbers understate it. On the IARD identity check, TT-VidT scores 70.99 versus 99%+ for baselines: it genuinely discards per-frame identity, a net loss for tasks that need actor consistency. On EPIC-Kitchens verb classification, which leans on objects and scenes, VideoMAE leads at 35.62 against TT-VidT's 32.54.