VLMs can think with value maps: latent visual tokens in CoT yield 20% accuracy gain

burny_tech · x · 2026-09-15

The author shares Scaffolding Minds, research from a Google DeepMind internship on multimodal latent reasoning.

Core idea: insert value maps directly into a VLM's chain of thought as latent visual tokens, letting the model "think with value functions" rather than just process images
Catch: off-the-shelf vision features aren't necessarily good "thoughts," so the latent representation itself is learned
This learned representation gives a 20% relative gain in average accuracy over a frozen encoder
Takeaway: what a model sees and what it should think with may be very different representations

More from Multimodal