Foresight lets streaming VLMs plan future perception without retraining, beating baselines by 9.5%

Ashok Prasad Neupane · hf · 2026-10-07

Foresight is a training-free method that lets streaming vision-language models dynamically plan future computation.

Key insight: streaming VLMs inherently anticipate the immediate future, which can guide computation without retraining.
A dual-stream Siamese LLM architecture (shared weights, encoders, KV cache) runs one LLM on the incoming stream while a second anticipates future context and plans when to reason, what to check, and how densely to sample.
Online reconfiguration uses schema-guided decoding and lightweight diff updates with minimal overhead.
With a frozen Qwen3-VL-8B backbone, it hits 23.0 mean joint F1 on OmniPro Online (beating the strongest trained baseline by 9.5%), plus +6.7 on StreamingBench and +15.4 on OVO-Bench. Code will be open-sourced.

More from Multimodal