Region tokens: an hour of video for fewer tokens than 20 seconds of patches
Cohere · youtube · 2026-11-14
A Cohere Labs talk by UIUC PhD student Savya Khosla on region-based visual representations:- Patch-based encoders quadruple token count when resolution doubles (even for uninformative pixels) and add hundreds of tokens per video frame even when frames barely differ.- Region tokens encode each meaningful entity as a single token persisting across frames, so compute scales with scene content rather than resolution or length.- Result: an hour of video uses fewer tokens than a patch encoder needs for a 20-second clip, while improving segmentation, retrieval, episodic memory localization, and spatial reasoning.The talk covers efficient region token generation, extension to video, and language alignment, targeting all-day perception for robots and wearables.
More from Multimodal
- Uncensored Qwen-Image-2.1 quantized model trends on Hugging Face — chatrlabs · 2026-11-14
- MiniMax H3 Image editing tested at 60MP: concept art to realistic photo in ~2 min on RTX 4090 — daroamer · 2026-11-14
- AI-generated 4D Gaussian Splatting brings Scooby-Doo's Velma to life — buchachos · 2026-11-14
- Encoder-free multimodal LLMs re-learn encoder-like traits in early layers — rdesh26 · 2026-11-14
- Hobbyist uses Grok Ultra and Google models to let AI design its own cyborg arm — Promptmethus · 2026-11-14
- Midjourney now lets you blend up to 20 style-reference images, and anime mixes look striking — Kyrannio · 2026-11-14