VoT: Vision-of-Thought for Unified Multimodal Representation Alignment
Jingxiang Sun, Chao Liao, Zhengxiong Luo, Chaorui Deng, Chen-lin Zhang, Junke Wang, Ceyuan Yang, Haoqi Fan, Weilin Huang
cs.CV, cs.AI, cs.CL
2026-09-08
ByteDance Seed inserts autoregressive visual-thought tokens between a frozen VLM and a DiT; the 21B model scores 0.91 on GenEval, above Mogao at 0.89 and FLUX.1-dev at 0.82.
Most text-to-image stacks still run "text encoder plus diffusion decoder": a prompt becomes a static embedding or KV cache that modulates continuous noise. That works for simple captions. The structured world knowledge sitting inside a VLM does not travel well through static conditioning, so the model can know what to draw and still render it wrong.
The other unification path tokenizes images with VQ-VAE / VQGAN-style pixel codes and predicts them autoregressively. Those codebooks optimize reconstruction, so tokens are long, local, and statistically heavy-tailed, poorly matched to a VLM's semantic space. ByteDance Seed's Vision-of-Thought (VoT) inserts a discrete layer between a frozen VLM and a diffusion branch: the VLM plans objects and layout as readable visual tokens, then the DiT paints pixels.
Training has three stages.
Stage 0 learns the tokenizer. Images go through a frozen Qwen2.5-VL-7B ViT, a one-layer trainable adapter, SimVQ into a 65,536 by 128 codebook, and a 6-layer decoder that reconstructs teacher ViT features. Three losses run together: feature reconstruction (keep visual information), VLM alignment (feed quantized visual embeddings into the frozen VLM and train caption cross-entropy so the tokens stay readable), and the usual VQ codebook plus commitment terms. Reconstruction alone yields compressed pixel shards; alignment alone drops detail. The paper treats them as complementary.
Stage 1 grafts a VoT branch onto Mogao-14B, a Mixture-of-Transformers backbone. Three pathways (text, VoT, diffusion) keep separate LayerNorm, QKV, and FFN, and concatenate KVs for shared attention. Text and VoT use causal masks; diffusion uses bidirectional attention. VoT experts are copied from pretrained VLM experts; the VLM branch stays frozen. Each image is 1,024 VoT tokens over a 65,536-way codebook.
Stage 2 jointly trains VoT and DiT with autoregressive cross-entropy plus diffusion velocity prediction. For classifier-free guidance, VoT tokens drop with probability 0.5, and text drops with probability 0.1 when VoT is present, giving roughly 50% text-only, 45% text-plus-VoT, and 5% VoT-only samples. A random 0-20% of VoT KVs are masked so the diffusion branch cannot treat the plan as a crutch. The finished model is 21B parameters (14B plus a 7B VoT branch). The appendix budgets 480,000 GPU-hours: 360,000 for pretraining (75%), 64,000 for SFT, 6,000 for RLHF.
On public GenEval (553 prompts, four images each) VoT scores 0.91 overall, above generation-only FLUX.1-dev at 0.82 (with an LLM rewriter) and unified models Mogao at 0.89 and BAGEL at 0.82. The gain sits in compositional slots: counting 0.86 vs Mogao 0.83, color attributes 0.82 vs 0.80, position 0.85 vs 0.84. Single-object is 1.00 on both, so the spatial gap is thin.
Internal AutoEval makes the tokenizer ablations readable. Mogao baseline 53.12. Alignment-only with a 2B teacher: 58.67. Add reconstruction and a 1-layer encoder: 64.64. Switch the teacher to Qwen2.5-VL-7B: 69.32. Six-layer decoder (Exp.E): 70.37. SFT plus RLHF (Exp.F): 76.22. A stronger teacher plus reconstruction beats chasing a lower autoregressive loss: Exp.A has the lowest VoT AR loss and the worst AutoEval.
Figure 4's internal instruction-following chart puts VoT at 76.22, GPT-Image-1 at 79.69, Seedream 4.0 at 78.2, and Gemini 2.5 at 69.24. The prose claims VoT outperforms Seedream 4.0 and GPT-Image-1; the bars do not. The number worth citing outside the lab is still GenEval, +0.02 over Mogao.
This is an incremental unification paper with a clean cut. Query-based conditioners such as MetaQuery and BLIP-3o are parallel and cheap at inference; they do not give the VLM a same-family, step-by-step visual plan. VoT makes that plan a trainable discrete layer, splits planning from rendering, and closes the tokenizer loop so the VLM can read its own visual tokens. Three pieces transfer: quantize teacher ViT features rather than pixels, train alignment and reconstruction together, and clone experts while freezing the VLM to cap cost.
The bill is also clear. 21B parameters and 480,000 GPU-hours are not a config tweak. Freezing the VLM keeps understanding intact and locks the system into one-way plan-then-paint.
The appendix says the frozen-VLM MoT design only transfers reasoning into generation. Bidirectional joint training is left as future work because of catastrophic forgetting.
The public GenEval lift is small, 0.89 to 0.91, with no variance or multi-seed report. AutoEval is unreproducible, and the figure disagrees with the caption. Training data is described only as a large text-image set. Understanding-side regressions are barely measured. The "interpretable" 1,024 tokens are not decoded back into objects or layouts in any quantitative study. RLHF is 1.3% of compute and still moves AutoEval from 70.37 to 76.22, so the planning layer and post-training are entangled.