Researchers show VLMs plan via image pointers before generating a single token
gsarti_ · x · 2026-11-20
Researchers at Northeastern's Bau Lab (Rohit Gandikota, David Bau et al.) opened up a VLM to see how it decides what to describe first, next, and when to stop.
Key findings:
- VLMs plan before generating a token: they first locate where the answer lies in the image and keep a pointer to that spot;
- The pointer is passed along to the token where the answer will be written, where dedicated attention heads follow it back to fetch the image content;
- The same pointers track what has already been said — each answer ties back to its source region, and before writing the next one the model reviews these spots and moves to the nearest undescribed location;
- The plan isn't an internal word list but a set of pointers into the image, re-read each step like whiteboard notes;
- Code LLMs and image diffusion transformers show similar planning behavior.
An interactive demo and source code let you watch the plan live.
More from Research
- alphaxiv launches preprint server where 936 AI agents authored 936 papers in 12 weeks — SRSchmidgall · 2026-11-20
- Composing EBM models at inference time: AND, OR, NOT with energy arithmetic — daibond_alpha · 2026-11-20
- Bo Dai and Yilun Du release EBM tutorial unifying ML through one energy-based lens — daibond_alpha · 2026-11-20
- EBM tutorial thread: ML learning methods differ only in negative sampling — daibond_alpha · 2026-11-20
- EBM tutorial's master table maps GANs, diffusion, BERT, and RL into one framework — daibond_alpha · 2026-11-20
- EBM tutorial thread: best-of-N and RL fine-tuning share one KL-regularized objective — daibond_alpha · 2026-11-20