CMU's Visual Token Routing cuts VLM agent visual tokens up to 5.8x with no accuracy loss
wellecks · x · 2026-11-05
Pranjal Aggarwal and Sean Welleck at CMU introduce Visual Token Routing (VTR), a method that slashes visual token overhead for computer-use VLM agents.
- Motivation: every screenshot costs thousands of visual tokens and agents add a new screen every step, so context balloons quickly.
- Method: a three-layer MLP router (23.3M params, 1% of the LM) reads token positions across recent frames and outputs keep probabilities; only needed tokens enter the VLM, and every 10th screenshot is kept in full. Trained on Qwen3.5-2B for just 2,000 steps.
- Results: matches full-resolution accuracy with 2.5x fewer tokens on OSWorld (2,040→808 per image), 2.9x on CUA-World, and 5.8x in video games; inference cost drops 1.8–3.3x and per-step prefill is 2.2–2.9x faster.
- Code and router weights are open-sourced.
More from coding & agent
- Levels decompiles Battlefield 1942 with Claude Code, ports it to browser with working multiplayer — huangyun_122 · 2026-11-05
- Kubernetes promotes AI Conformance to permanent SIG Architecture subproject — TerryTangYuan · 2026-11-05
- quiet.md goes open source: Rust+GPUI editor, Feishu-compatible, ~5ms frames on 1M-char docs — dotey · 2026-11-05
- A background agent that auto-structures messy scientist Excel data, day 17 of agent series — nlarusstone · 2026-11-05
- Code mode isn't about MCP: it's about whether tool results go to context or program variables — ritakozlov · 2026-11-05
- Agent Arena: frontier models show 55-minute gap on long-horizon agent tasks — arena · 2026-11-05