Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning
Xiaoyu Zhu, Xinke Deng, Suresh Taddewadikar, Arnab Kumar Mondal, Zhongyu Jiang, Ian Fasel, Joerg Liebelt
cs.CV, cs.AI, cs.CL, cs.LG, cs.MM
2026-08-17
Apple's IVT trains video models to predict future-frame latents alongside answers, then answers directly at inference: it beats text-only SFT on all 6 settings and runs 5.4x faster than Visual CoT.
Proactive video reasoning asks a model to speak before the event is over: early-event prediction (EEP, identify an ongoing action from a 10-70% partial observation) and next-event prediction (NEP, describe an action that has not started yet). A prediction that arrives late is worthless, so latency matters as much as accuracy.
The two existing routes both fall short. Answer-only fine-tuning maps partial video straight to text and never models the future. Visual CoT generates an explicit future frame and conditions the answer on it. The idea is intuitive but expensive: decoding an image into visual tokens and re-encoding it back pushes mean latency from 1.20 s to 6.56 s per sample, a 5.5x increase.
The paper's diagnostic experiment is the sharpest part. Swap the model-generated future frame for the ground-truth one and early-event ROUGE-L jumps 21.6%, while the generated frame only buys 5.9% and even hurts next-event prediction. The bottleneck is not thinking about the future; it is the cost and fidelity of rendering that thought as pixels.
IVT (Internalized Visual Thinking) moves the imagining into training. On unlabeled video, for each observed prefix, frames 1 to H seconds ahead (sampled at 1 fps) are encoded by a target encoder (Flux-VAE in the best configuration) into M latent tokens. The model produces hidden states at designated prediction positions, a projection head maps them into the target space, and a prediction loss supervises them, with stop-gradients on the target side. Text answer prediction runs alongside; the total loss is Ltext + Lpred with equal weights. At inference the model runs the exact Answer-Only SFT computation graph: video in, text out, no image generation.
The backbone is BAGEL-7B (Apple's unified understanding-generation model, about 14B total parameters), trained for 20k steps. The substance of the paper is a systematic ablation over six design axes: target representation (Flux-VAE latents, DINOv2, SigLIP2 adaptive or frozen), decoder (shared Dense vs MoE with per-token-type experts), prediction horizon (H=1 to 4), data mixture (understanding-to-prediction at 1:1, 3:1, 5:1), curriculum (joint vs two-stage), and objective (direct regression vs rectified flow matching).
Across Ego-Exo4D, Ego4D and EPIC-KITCHENS-100, IVT beats Answer-Only SFT on all four metrics in all six dataset-task settings. Against Visual CoT under matched backbone and data (ROUGE-L):
| Setting | Answer-Only SFT | Visual CoT | IVT best |
| Ego-Exo4D EEP | 28.8 | 32.7 | 33.3 (MoE) |
| Ego4D EEP | 33.3 | 36.1 | 34.8 (Dense) |
| EPIC EEP | 36.5 | 40.2 | 39.0 |
| Ego-Exo4D NEP | 46.3 | 42.6 | 48.5 (MoE) |
| Ego4D NEP | 51.9 | 52.2 | 52.7 (Dense) |
| EPIC NEP | 45.5 | 42.8 | 49.1 (MoE) |
Visual CoT keeps a 1.2-1.3 point lead on two early-event splits, but IVT wins all three next-event settings; averaged over the three benchmarks, Dense IVT lifts NEP ROUGE-L from 45.9 to 49.7 over Visual CoT. Text-CoT distilled from Qwen3-VL-30B trails plain SFT almost everywhere, collapsing to 32.3 against 45.5 on EPIC NEP.
Latency is structural: on one B200 at batch size 1, Visual CoT averages 6.56 s per sample with P95 at 7.92 s, while IVT averages 1.22 s with P95 at 1.77 s, nearly identical to the 1.20 s / 1.92 s of Answer-Only SFT.
On out-of-domain transfer, with Charades held out from training entirely, SFT reaches 71.7% accuracy, Visual CoT drops to 58.2%, and Dense IVT is the strongest at 73.2%.
The most counterintuitive ablation: two-stage training (10k steps of prediction, then task-only fine-tuning) lands below the no-prediction baseline, 28.6 vs 30.6 ROUGE-L, while joint training reaches 34.5 (+12.6%). Predictive supervision has to stay coupled to the text objective; split it into sequential stages and the learned representations are lost during fine-tuning. Other findings: Flux-VAE latents are the strongest target, with DINOv2 semantic features helping less; a 1:1 data mixture leads across the board at 20k steps while 3:1 and 5:1 rise fast early then plateau; the Dense decoder wins at short horizons (37.4 vs 33.8 for MoE at H=2), while MoE is less horizon-sensitive and peaks at H=3.
For teams building streaming or proactive vision systems (glasses assistants, robots, monitoring), this is a replicable recipe for converting visual foresight from an inference cost into a training cost. The inference graph does not change, deployment stays as is, and the gains come entirely from post-training. It also answers a live question about unified models: the value of a generative branch may lie less in producing good images than in learning predictive representations.
The paper is unusually clear about failed recipes too. Six design axes, each with measured sensitivity, so a team adopting the approach can avoid plausible-looking traps such as two-stage curricula.