Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs
Youngwoo Shin, Yusung Ro, Minseo Kim, Junmo Kim
NeurIPS 2026
cs.CV, cs.AI
2026-10-01
VideoLLM temporal signals peak at mid-layers then fade toward the output; training-free reinjection of the peak difference lifts temporal reasoning on 3 models and 4 benchmarks.
VideoLLMs share a well-documented blind spot: reverse the frame order of a video and the correct answer to a temporal question should flip, yet the model often predicts the same thing. Existing fixes work around the failure from outside. ArrowRL trains with dedicated temporal rewards; contrastive decoders like TCD and DINO-HEAL subtract output distributions obtained from temporally distorted inputs. Neither asks where inside the model the failure happens. The KAIST team did, using logit lens (projecting each layer's hidden state onto the output vocabulary) to trace answers layer by layer. On reversed videos the correct answer briefly holds the top probability around the middle layers, then later layers overturn it. The model does extract temporal information; it just fails to keep it. That reframes the problem from "cannot do temporal reasoning" to "where does the signal get lost".
Diagnosis first. For each video, build a reversed copy and take the last token's hidden state at every layer (under causal attention, the only position that has seen all visual and text tokens). The forward-minus-reversed difference is the temporal divergence vector τl; its normalized magnitude τ̂l, plotted per layer, gives the temporal divergence profile. Three architecturally distinct models (Qwen2.5-VL-7B, Qwen3-VL-8B, InternVL2.5-8B) produce the same shape: near-zero divergence early, a sharp peak in mid-to-late layers, then progressive decline toward the output.
Two controls pin the peak to temporal reasoning. Keep the video pair fixed and swap only the question: temporal questions ("is the person moving downwards?") show the peak, while spatial ("indoor or outdoor?") and video-irrelevant ("does a week have seven days?") questions flatten the curve. Attention knockout, which blocks the last token from attending to earlier positions at a given layer, causes the largest drop in ground-truth probability exactly at the peak layers, so that signal is causally used.
The intervention, Temporal Activation Injection (TAI), follows the measured decay:
| Model (baseline → TAI) | TempCompass | TVBench | AoTBench |
| Qwen2.5-VL-7B | 73.4 → 75.6 | 44.6 → 46.6 | 62.9 → 63.7 |
| Qwen3-VL-8B | 77.1 → 78.0 | 49.9 → 52.6 | 67.4 → 67.6 |
| InternVL2.5-8B | 70.9 → 73.6 | 57.2 → 58.3 | 71.6 → 71.6 |
Against baselines: on Qwen2.5-VL-7B, TAI's TempCompass 75.6 beats TCD (74.2), DINO-HEAL (72.7) and even ArrowRL (75.0), which requires reinforcement learning; on TVBench its 46.6 sits just under ArrowRL's 46.9.
The specificity results carry more weight than the averages. Flipping the sign of β (Anti-TAI) selectively crashes the reversal-sensitive categories: Attribute Change -36.3, Order -27.4, Direction -12.6, while Action (-0.4) and Speed (-0.7) barely move. Normal TAI concentrates gains the same way (Order +3.9, Attribute Change +6.6), at the cost of 1.7 points on Action for Qwen2.5-VL-7B. General capability holds: MVBench rises from 54.5 to 57.7 on the same model. TAI also stacks on ArrowRL-trained weights, adding +1.5 on TempCompass and +0.9 on TVBench. On schedules, profile weights (75.6) edge out uniform injection (75.4) and a reversed schedule (75.1); β beats the baseline across the full 0.1-1.0 range and plateaus after 0.5.
The τ̂l profile is a deliverable in itself: annotation-free, computable from one forward pass, applicable to any VideoLLM that exposes intermediate layers. TAI needs no training, code is released, it runs on a single A6000, and its overhead is one extra forward pass truncated at Lsrc per input. It can be dropped onto existing models, including RL-trained ones. The gains, mostly 1-3 points, are incremental; the value lies in diagnosing the failure mechanism before intervening, and in near-zero integration cost. The broader lesson generalizes: what a model computes in its middle layers does not necessarily survive to the output, and measure-then-compensate is a playbook worth trying on other capability axes.
The authors scope the work to frame-order temporal information; spatial reasoning, object interaction and audio-visual correspondence are out of scope, with other contrastive axes left to future work. A few caveats from the tables: "consistent improvement across all three benchmarks" breaks on AoTBench with InternVL2.5-8B, where the average stays flat at 71.6 and only the temporal-relevant subtask moves +0.2. Every input needs an extra reversed forward pass; although it exits early, the paper never quantifies the latency overhead. β is tuned per model, and the profile is built from short TempCompass clips under uniform 16-frame sampling, so long videos and real film footage are untested. Action on Qwen2.5-VL-7B drops 1.7 points, so "negligible impact on non-temporal tasks" deserves a per-category asterisk.