Apple's Internalized Visual Thinking matches VisualCoT accuracy with ~5x faster video reasoning
机器之心 · wechat · 2026-09-03
Apple researchers propose Internalized Visual Thinking (IVT), a post-training framework for proactive video reasoning that learns to predict the future internally instead of generating future frames at inference time.
Motivation
- Explicit VisualCoT lifts early/next-event prediction accuracy but pushes latency from 1.1-1.3s to 6.5s per sample, with unstable gains. Oracle experiments with ground-truth future frames confirm the information is valuable (ROUGE-L 34.4 → 41.9), so the bottleneck is costly, error-prone pixel-level generation.
Method
- IVT adds a Next-Embedding Prediction objective during training: the model predicts latent embeddings of future frames alongside text answers, forcing it to capture motion trends, state changes and interaction intent. The prediction branch is removed entirely at inference.
- Key findings: Flux-VAE features work best as future targets (fine-grained spatial detail matters); joint training with the language objective beats two-stage training; short prediction horizons work best, longer ones add noise.
Results
- IVT beats Answer-Only SFT on all 6 dataset-task settings across Ego-Exo4D, Ego4D and EPIC-KITCHENS-100, and beats VisualCoT on 4 (all three next-event benchmarks).
- Average inference: 1.22s vs 1.20s for SFT — a 5x speedup over VisualCoT's 6.56s, with no deployment cost increase.
Limits: no gains at longer hop=3 horizons; multi-branch futures remain hard to evaluate. IVT opens a middle ground between explicit visual chain-of-thought and pure text reasoning: think during training, answer directly at inference.
More from Research
- Flow Reasoning Models repo out: stable attractors solve Sudoku-Extreme with 44× fewer FLOPs — alec_helbling · 2026-09-03
- Flow Reasoning Models: recurrent flow architecture hits 99.5% on Sudoku-Extreme with 44x fewer FLOPs — alec_helbling · 2026-09-03
- Astra's n^1/400 approximation hardness could enable crypto assuming P≠NP — thomasahle · 2026-09-03
- AI reads chest CTs to score each organ's age, validated on 35,000 scans — rand_longevity · 2026-09-03
- Eigenfaces nostalgia: CV professor revisits the technique that defined early face recognition — CSProfKGD · 2026-09-03
- HCOMP 2026 + CI 2026 joint conference registration open; student early-bird at $300 — windx0303 · 2026-09-03