CoLT: Multimodal Latent Chain of Thought Reasoning
机器之心 · wechat · 2026-07-15
This article introduces CoLT (Chain of Latent Thoughts), a multimodal latent space reasoning method. It aims to enable MLLMs to reason using a few continuous latent vectors instead of verbose text CoT, without requiring extra image annotations.
Core Approach
- Uses K=3 latent thought vectors to progressively unfold the reasoning. Each step utilizes the last-layer hidden state at a specific position in the language model, feeding it directly back into the model.
- Training incorporates triple step-level supervision: forward decoding (latent vectors reconstruct the next text reasoning step), backward decoding (semantic alignment with preceding reasoning), and inter-step prediction (ensuring coherence between adjacent latent vectors).
- Decoders and projection heads are discarded during inference, retaining only the 3 latent vectors, resulting in almost zero extra inference overhead.
Key Results
- Achieves an average accuracy of 79.1% across 8 multimodal benchmarks, outperforming TextCoT (75.7%) with the same backbone, as well as existing latent reasoning and latent visual reasoning methods.
- Compared to TextCoT's average of 142 reasoning tokens, CoLT uses only 3 latent vectors, achieving a 22.6× speedup in text decoding and a 10.1× end-to-end inference speedup.
- Excels in tasks like MMbench and ChartQA, showing significant gains in chart and OCR scenarios.
Ablation and Interpretability
- All three supervision signals are effective, with backward decoding contributing the most.
- A small external decoder like Qwen3-0.6B is sufficient; scaling up yields minimal gains.
- The optimal number of latent thought steps is K=3; fewer leads to insufficient capacity, while more introduces redundancy.
- By mapping latent vectors back to natural language via the training decoder, it demonstrates that latent thoughts aren't black-box noise but carry identifiable reasoning fragments.
More from Research
- WeirdChat catalogs strange model behaviors from more than 100 million sampled responses — JacobSteinhardt · 2026-07-22
- New agentic benchmark shows AI managers escalate to coercion and fake success — Jasmine Brazilek · 2026-07-22
- Ai2’s Asta adds one-click handoff and self-checking deep paper search — allen_ai · 2026-07-22
- DepthART pushes monocular depth to tiny models at 1000 FPS on RTX A6000 — kwangmoo_yi · 2026-07-22
- Meta says SAM 3 and DINOv3 cut 3D volume labeling from a month to 15 minutes — AIatMeta · 2026-07-22
- Project CETI gets a Jeopardy! shout-out with a SETI-style whale clue — begusgasper · 2026-07-22