LVMT: Video Mask Transformer for Long-term Video Segmentation
Narges Norouzi, Niccolò Cavagnero, Idil Esen Zulfikar, Bastian Leibe, Gijs Dubbelman, Daan de Geus
cs.CV
2026-09-28
LVMT swaps PMT's query sum for a GRU memory and trains 15-frame clips with truncated backprop, lifting OVIS AP from 51.8 to 56.4 at 95 FPS, over 10× faster than DVIS-DAQ.
Online video segmentation has to emit a mask and a class on every frame, and keep one identity for each instance across the clip. VidEoMT and its follow-up, PMT, reduce that pipeline to a pretrained ViT plus a set of queries carried forward in time. A query is the vector for one object slot. PMT freezes the ViT and trains only a light decoder, so the same encoder can be reused across tasks. Long occlusions still break identities.
The query handed to the next frame is a linear projection of the previous output, added to learnable queries that ignore time. Those learnable queries exist so new objects can be detected. While an object is hidden, the encoder has no new evidence to write, but the learnable vector is added again every frame. The stored identity thins out. When the object returns, the slot no longer matches it. That error is an identity switch.
An explicit bank of past queries, the GenVIS route, can hold occluded objects. Compute and memory then grow with the video. Training is short too. PMT's default clip is 5 frames. After a GRU is inserted, the 20 videos with the most identity switches have a mean occlusion of 13 frames, 2.6 times the training length, about 3 times as many objects, and about 5 times as many occlusion events as the videos with the fewest switches. Naively training on 10 frames drops AP by 1.9. The paper attributes that drop to gradients weakening across more recurrent steps. Longer clips run out of memory.
LVMT keeps the PMT stack. A frozen ViT, ViT-L initialized from DINOv3 by default, produces patch features. A Plain Mask Decoder turns those features and the queries into segmentation queries, which a small head reads as a class and a mask. Query addition is replaced by one GRU cell shared across slots. The hidden state has a fixed length, so cost does not grow with the video.
Frame 0 has no history. The learnable queries initialize both the decoder input and the GRU state. On later frames the GRU reads the current segmentation queries and the previous state, and writes the queries that enter the next frame. If an object is absent, the gates can write little and keep the old state.
Longer supervision uses Truncated Query Propagation (TQP), truncated backpropagation through time applied to queries. A clip of T frames is split into chunks of at most F frames. The default is F=5 and T=15, three chunks. Chunks run in order inside one optimizer step. The final GRU state crosses each boundary in the forward pass and is detached, so the next chunk sees the history and the gradient does not. The step uses the average of the per-chunk gradients. Peak memory stays at one chunk. Inference is unchanged.
Each object is matched to a query on the frame where it first appears, and that assignment is kept. Training uses AdamW and mixed precision, with the same batch size, learning rate, resolution, and iteration count as PMT.
OVIS val is built to stress heavy occlusion and crowded scenes. The table is the mean of five ViT-L runs. FPS is measured on one H100 at batch size 1 with FlashAttention v2 and torch.compile.
| Setup | OVIS AP | Params | GFLOPs | FPS |
| PMT, 5-frame clips | 51.8±0.3 | 358M | 1014 | 97 |
| GRU only, 5 frames | 54.2±0.4 | 363M | 1015 | 95 |
| GRU, 10 frames, full backprop | 52.3±0.3 | 363M | 1015 | 95 |
| GRU+TQP, 10 frames | 56.1±0.3 | 363M | 1015 | 95 |
| LVMT, 15 frames | 56.4±0.4 | 363M | 1015 | 95 |
The GRU alone adds 2.4 AP. On 10-frame clips, full backpropagation drops AP from 54.2 to 52.3, while TQP reaches 56.1, 3.8 above that naive long clip. At 15 frames, LVMT is 4.6 above PMT's 51.8±0.3. Parameters move from 358M to 363M, GFLOPs from 1014 to 1015, and FPS from 97 to 95.
Smaller backbones gain more, at speeds described as close to the matching PMT. Exact FPS for those sizes is not listed. ViT-B goes from 42.7±0.5 to 48.4±0.2 (+5.7). ViT-S goes from 31.5±0.4 to 39.3±0.3 (+7.8).
The external table does not report a mean over runs. The frozen ViT-L rows below use DINOv3. DVIS-DAQ uses DINOv2 and fine-tunes the encoder.
| Method | OVIS AP | FPS | YT-VIS 2022 AP | FPS |
| DVIS-DAQ | 54.3 | 8 | 42.0 | 10 |
| CAVIS | 53.4 | 9 | 42.2 | 13 |
| VidEoMT | 51.9 | 104 | 42.8 | 137 |
| PMT | 52.0 | 97 | 45.8 | 123 |
| LVMT | 56.7 | 95 | 51.5 | 121 |
On OVIS, LVMT is 2.4 AP above DVIS-DAQ, at 95 FPS against 8, more than 10 times the frame rate. On YouTube-VIS 2022 it is 9.3 above CAVIS and 8.7 above VidEoMT. LOMM reports 48.2 AP at 12 FPS. With DINOv2 on both sides, LVMT scores 48.4 AP at 127 FPS, so accuracy is essentially tied.
On tracking metrics, identity switches fall from PMT's 2755 to 2001, about 27% fewer. IDF1 moves from 78.9 to 81.8, and AssA from 73.3 to 78.4.
For panoptic segmentation on VIPSeg with DINOv3, LVMT reaches VPQ 60.3 and STQ 53.4 at 57 FPS, against PMT at 55.5, 49.2, and 58 FPS. Semantic segmentation on VSPW moves less: mIoU from 65.7 to 66.4, mVC from 94.9 to 95.3. Against VidEoMT with DINOv3, mIoU is 2.4 higher (66.4 vs 64.0) and mVC is 0.9 higher.
Memory choices, as AP/FPS: GRU 56.7/95, LSTM 56.0/93, S5 55.8/96, DeepGRU 56.5/91, explicit memory bank 54.5/86. The GRU is 2.2 AP above the bank and slightly faster.
A multi-scale ablation uses a different resolution schedule, so its absolute AP does not match the table above. At 10 frames the four cells are 51.1 (neither), 52.6 (GRU only), 54.5 (TQP only), and 56.5 (both). At 15 frames, training without TQP runs out of memory; both pieces together score 56.7. At 20 frames the score is 56.8, a gain of 0.1. Mean occlusion on this set is already 13 frames, so the extra window has little left to cover.
Deployment cost stays in PMT's range. The ViT remains frozen and reusable across instance, panoptic, and semantic segmentation. The GRU state stays fixed, and attention does not turn into a longer sequence as the video grows.
The useful case is online instance or panoptic segmentation where identities must survive occlusion. Against DVIS-DAQ, which fine-tunes a ViT-Adapter, the H100 protocol is about 10 times the frame rate, with higher OVIS AP. Semantic segmentation gains 0.7 mIoU over PMT, a weak reason to switch. The change is in propagation and in the training window. The visual backbone is still a frozen ViT.
Extending training from 15 to 20 frames adds 0.1 AP. That matches a mean occlusion of about 13 frames on OVIS: the window already covers a typical gap. The paper says longer occlusions may benefit more, and the main text does not test such data. Chunking bounds training memory. It does not show that a fixed state still holds an identity over minute-long video.
TQP cuts gradients at chunk borders. A loss in a later chunk does not revise writes made earlier. The forward pass sees 15 frames, while credit assignment still runs in blocks of 5. There is no separate evidence that the long-range write itself is optimized.
On VIPSeg with DINOv2 for both models, DVIS-DAQ still leads, at VPQ 57.4 and STQ 52.0 against LVMT at 56.6 and 51.3. The stated gain of 2.9 VPQ compares DINOv3 LVMT (60.3) with DINOv2 DVIS-DAQ (57.4). On OVIS with DINOv2 for both, LVMT at 55.6 is only 1.3 above DVIS-DAQ at 54.3. The headline gap of 2.4 mixes in a newer encoder initialization.
On VSPW the gain over PMT is 0.4 mVC and 0.7 mIoU. The five-run standard deviation on the main ablation is about ±0.4, and the large comparison tables do not report repeated runs. Gaps under one point are soft. Scores on YouTube-VIS 2019 and 2021 appear only in the appendix. The main text says those gaps are smaller because the sets are easier.