VLMs can think with value maps: latent visual tokens in CoT yield 20% accuracy gain
burny_tech · x · 2026-09-15
The author shares Scaffolding Minds, research from a Google DeepMind internship on multimodal latent reasoning.
- Core idea: insert value maps directly into a VLM's chain of thought as latent visual tokens, letting the model "think with value functions" rather than just process images
- Catch: off-the-shelf vision features aren't necessarily good "thoughts," so the latent representation itself is learned
- This learned representation gives a 20% relative gain in average accuracy over a frozen encoder
- Takeaway: what a model sees and what it should think with may be very different representations
More from Multimodal
- Audio8-TTS-Realistic-Air, a zero-shot voice-cloning TTS model, trends on Hugging Face — Edge0 · 2026-11-02
- Mistral explains why its model-built solver is a first step toward new drugs and batteries — MistralAI · 2026-11-02
- Mistral Large 4 writes a quantum chemistry SCF solver from scratch with one short prompt — MistralAI · 2026-11-02
- Gorgeous dithered pixel art collage serves up a 'great morning' — philipvollet · 2026-11-02
- Clown Prophecy moves to new repo, now uses first clip as reference to fix H3 quality drift — reeight · 2026-11-02
- A 25-year design veteran asks: who still mixes pro software with AI tools? — Opening-Knee-5913 · 2026-11-02