RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation
Jaeyeon Heo, Geuntaek Lim, Yukyung Choi
cs.CV
2026-10-14
RTAformer injects calibrated adjacent-frame relation-query deltas. On Action Genome, matched ViT-B/14 SGDET R@10 gains 4.8 on transition frames vs OED, and 1.7 elsewhere.
Video scene graph generation (VidSGG) writes, for every frame, the objects and a triple of subject, predicate, and object. A person and a cup can both stay visible while looking at becomes not looking at on the next frame.
Packing several frames into one temporal context steadies the predictions. It also makes them stick to the previous predicate. That failure is relational semantic inertia: the pair is unchanged, the relation has moved, and the model still emits the old label. In Figure 1, OED keeps predicting looking at after the ground truth has switched to not looking at. Reimplemented with a frozen DINOv2 ViT-B/14, a self-supervised vision backbone, that baseline scores SGDET R@10 of 45.9 on non-transition frames and 34.3 on transition frames under With Constraint, where each pair keeps only its top predicate. The gap is 11.6 points.
RTAformer is a one-stage set predictor and does not call a tracker at inference. A frozen DINOv2 ViT and a Transformer encoder produce frame features. Each of 100 triplet queries stands for one candidate subject-object pair.
Each frame runs two decoders of six layers. The Instance-Pair Decoder lets the query read that pair's spatial context. The Relation Context Decoder then reads the relation conditioned on the pair. The predicate depends on who is there and where the boxes sit, so this order is fixed.
Temporal refinement is split. Disentangled Temporal Aggregation (DTA) runs an instance stream and a relation stream, three decoder layers each. Identities persist across frames; predicates can flip at once. One shared stream lets the evidence that the person is still present smooth away the relation change. Each layer keeps a smaller share of reference queries, at ratios 0.8, 0.5, and 0.3. The instance stream ranks those references by object-class confidence.
Before the relation stream aggregates across time, Transition-Aware Delta Injection (TADI) writes in a change cue from adjacent-frame relation queries. At a fixed query index, the current relation query minus the previous one is the displacement. It is split into an RMS magnitude rq and a direction normalized by rq. A two-layer MLP reads log(1+rq) and produces a bounded scale gate. That gate is multiplied by the geometric mean of object confidence on the two frames, so a poorly seen side updates the predicate less. A learned global strength adds the residual back onto the current relation query. Reference relation queries are also scored with a displacement-saliency term.
Same-index subtraction assumes the index still denotes the same pair. DETR decoding does not guarantee it. Spatio-Semantic Instance Alignment (SSIA) applies a symmetric InfoNCE loss during training only. InfoNCE is the contrastive loss that pulls a designated positive pair together. Here it pulls same-index instance-pair queries together when the previous frame contains an object of the same category at IoU above 0.3. Temperature is 0.1 and the loss weight is 0.1. The loss is absent at inference.
The schedule follows OED. Phase one freezes the backbone and trains spatial parsing plus SSIA. Phase two freezes the spatial modules and trains DTA and the temporal heads. Test-time inference runs spatial parsing, DTA, and classification in one pass, with no temporal post-processing.
Action Genome is the benchmark: 9,848 videos, 234,253 annotated frames, 35 object categories excluding person, and 25 predicates grouped into attention, spatial, and contacting. Under SGDET the model must produce boxes, classes, and predicates. With Constraint keeps one predicate per pair. No Constraint allows several. mR@K averages recall over predicate classes.
UNO, also a one-stage model on frozen DINOv2, is the matched comparison. With Constraint R@10 goes from 36.7 to 37.7 on ViT-S/14, 38.2 to 42.2 on ViT-B/14, and 39.3 to 44.6 on ViT-L/14. No Constraint R@10 gains are 2.5, 4.9, and 6.4. ViT-L/14 reaches R@10/20/50 of 44.6/52.9/60.2 with constraint and 47.2/57.8/65.9 without, the highest R@K in Table 1. It also posts the highest reported mR: 30.5/38.3/44.2 with constraint and 37.2/53.5/64.4 without. ARN on ResNet-50 reports with-constraint mR@10 of 23.2, and DIFFVSGG reports 24.7. UNO does not report mR.
| Method | Metric | Score | Comparison |
| RTAformer ViT-L/14 | WC / NC R@10 | 44.6 / 47.2 | UNO 39.3 / 40.8 |
| Matched backbone, transition | WC R@10 | 39.1 | OED reimpl. 34.3 |
| Matched backbone, other frames | WC R@10 | 47.6 | OED reimpl. 45.9 |
| rq as a transition score | AUROC | 0.621 | cq 0.442 |
On ViT-B/14, With Constraint R@10/20/50 is 39.7/47.5/53.7 for spatial parsing alone, 40.8/48.1/53.9 with SSIA only, 41.2/49.4/56.8 with DTA only, and 42.2/50.1/57.6 with both. With DTA enabled, SSIA still adds 1.0 R@10.
For the injection design, on top of spatial parsing plus SSIA, temporal aggregation without any delta lifts R@10 from 40.8 to 41.3. Direction-only injection reaches 41.7, raw displacement 41.9, and full TADI 42.2. The gates contribute, and each step is under a point.
Transition frames are built from persistent pairs. Of 105,356 adjacent pairs with IoU above 0.3 and a consistent category, 35,523 (33.7%) replace at least one predicate family with a disjoint set. The split is 25,883 transition frames and 21,474 non-transition frames. Matched-backbone OED scores With Constraint R@10 of 34.3 versus 45.9. RTAformer scores 39.1 versus 47.6, so the gains are 4.8 and 1.7. Under No Constraint the gains are 4.1 and 1.9. Parameters are 123.8M against 126.8M. The text says GFLOPs and speed stay comparable and does not give a frame rate.
Released OED and TEMPURA checkpoints use different backbones. On transition frames RTAformer is still ahead of both, by 8.0 and 8.6 R@10 points versus OED under the two constraint settings. Those figures should not be added to the matched-backbone 4.8.
Larger rq lines up with more transitions. Across deciles of equal count, the transition rate rises from 20.9% to 55.4%, compared with 33.7% over all persistent pairs. Mean rq is 0.396 on transitions and 0.289 otherwise. As a score with no extra classifier, rq reaches AUROC 0.621, and 0.633, 0.636, and 0.632 inside the three predicate families. Foreground confidence cq is lower on transitions, 0.517 versus 0.591, and its AUROC as a transition score is 0.442. Magnitude carries transition information. The confidence gate limits rewrites on blurry queries.
Full-set R@K mixes pairs that stay easy because the objects never left with pairs whose predicate actually changed. After the split, matched-backbone gains are 4.8 on transition frames and 1.7 on the rest. Navigation and video QA should be judged on the transition subset.
The part to reuse is the calibrated relation-query delta, with the instance stream left in charge of identity. Inference adds no branch and does not need the text signal TR2 uses to supervise inter-frame relation differences. Against UNO, With Constraint R@10 is only 1.0 higher at ViT-S/14 and 5.3 higher at ViT-L/14. On the small backbone this is a modest step.
The paper states two limits directly. Same-index queries often keep the same pair in qualitative checks, yet DETR decoding does not guarantee cross-frame alignment, so the displacement can include slot drift. Tests cover Action Genome under SGDET only, with no second dataset and no other protocol.
Most multi-stage entries in Table 1 use ResNet-101, and the published OED and ARN numbers use ResNet-50. Beside DINOv2, the matched comparisons are UNO and the reimplemented OED. UNO reports no mR, so the class-balanced lead has no near one-stage check on the same setup. On full-set With Constraint R@10, spatial parsing alone already reaches 39.7, against 38.7 for matched OED. The temporal module's advantage on the full set is modest.
On the full test set, TADI moves R@10 only from 41.3 to 42.2 relative to temporal aggregation without injection. The 4.8-point transition gain is not broken down by component. rq separates transitions at AUROC 0.621. Two-phase training freezes the backbone and then the spatial modules, so the temporal head never trains jointly with spatial features. A counted transition also requires disjoint predicate sets. Partial overlap is excluded, and 33.7% is tied to that rule.