Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
Siting Li, Zhengyang Wang, Simon Shaolei Du, Xi Chen, Yang Liu
cs.CV, cs.CL
2026-09-09
On a Qwen3 pure-AR testbed, better rFID does not imply better joint modeling. I2T loss is the stabler cross-tokenizer signal; a DINO swap drops VQAv2 from 52.25 to 51.31.
Unified multimodal models put images and text under one autoregressive objective. The image tokenizer is the visual vocabulary of that model, yet people still pick tokenizers with rFID, ImageNet probes, or generation-only and understanding-only scores. None of those answers what happens when image tokens and text tokens are modeled jointly by the same network and the same next-token loss.
Amazon FAR and the University of Washington built a controlled pure-AR testbed and treated tokenizers as visual languages. They track task-wise validation losses as data and model size grow, relate those losses to downstream scores, and then ask whether discriminator type, semantic supervision, and vocabulary size change reconstruction, joint learnability, or both.
The backbone is Qwen3 (0.6B, 1.7B, and 4B for the main grid; 8B for a recipe check). The vocabulary is expanded with discrete image tokens plus <boi> and <eoi>. Images are 256×256 with K=16×16 tokens. Only single-codebook tokenizers are compared: IBQ at 1024 / 8192 / 16384, GigaTok and GigaTok-DINO, UniTok and UniTok-sem. Continual pretraining goes up to 60 million samples (6.6 million text, 53.3 million image-text, mixed 1:8). Of the image-text data, 80% is T2I and 20% is I2T, with 10% of T2I made unconditional. Loss is computed only on predicted tokens. Supervised finetuning then runs for two epochs on 4.9 million instruction samples.
Held-out sets of 50,000 text and 50,000 image-text samples yield four losses: text, unconditional image, T2I, and I2T. Downstream metrics are GenAI-Bench, MJHQ-30K, and post-SFT VQAv2 / GQA. As a recipe check, Qwen3-8B with the Chameleon tokenizer after 60 million samples scores GenAI 0.73, WISE 0.38, and mean VQA 60.83, close to Liquid-7B at 90 million samples (0.72 / 0.41 / 61.40), while MJHQ-30K gFID is worse (10.55 vs 5.47).
All four losses fall with more data and larger models, but the slopes differ and tokenizer rankings flip by task. Text loss is mostly inherited from the language-model init; image-side losses are what continual pretraining actually moves. T2I tracks unconditional image loss. I2T is also scored on text tokens, yet it does not follow pure-text loss.
With a fixed tokenizer, both T2I and I2T loss line up with generation quality. Across tokenizers, the T2I loss-quality mapping shifts with the image-token space. Dividing by log2(B) removes part of that shift; the rest tracks rFID. I2T, scored on a shared text vocabulary, is the more stable cross-tokenizer signal, and pre-SFT I2T still correlates moderately with post-SFT generation and general VQA. The link is weaker on OCR-heavy TextVQA.
Reconstruction and joint learnability can point opposite ways:
| Tokenizer | rFID↓ | Text↓ | I2T↓ | T2I↓ | GenAI↑ | VQAv2↑ |
| GigaTok-DINO | 0.51 | 2.949 | 1.660 | 7.437 | 0.720 | 51.31 |
| GigaTok | 0.81 | 2.937 | 1.661 | 7.416 | 0.720 | 52.25 |
| UniTok | 1.86 | 2.883 | 1.670 | 6.264 | 0.670 | 57.21 |
| UniTok-sem | 2.23 | 2.873 | 1.655 | 5.848 | 0.690 | 61.28 |
A DINO discriminator cuts GigaTok rFID from 0.81 to 0.51. The three losses barely move, GenAI stays at 0.720, and VQAv2 drops from 52.25 to 51.31. Adding semantic supervision to UniTok worsens rFID from 1.86 to 2.23, lowers all three losses, and lifts GenAI from 0.670 to 0.690 and VQAv2 from 57.21 to 61.28. Ablations pin the text-side hit on the image-token prediction objective, not on I2T. The I2T gain from semantic supervision concentrates on COCO object words rather than simpler local visual n-grams. IBQ vocabularies of 1024, 8192, and 16384 are non-monotonic in loss, but 16384 still wins downstream, which the paper ties to higher reconstruction fidelity.
Picking a tokenizer by rFID alone can discard a visual language that is easier to model jointly, and can even hurt text modeling under a shared objective. The practical signal is task-wise loss, especially I2T: it is comparable across tokenizers and already hints at post-SFT generation and general VQA. Semantic supervision, which makes reconstruction worse while making tokens more word-like, may be a better bet in unified AR than swapping in a DINO discriminator. This is a diagnostic, not a new SOTA generator.
The study covers seven fixed-length, single-codebook discrete tokenizers. Resolution and token count K are held fixed, because changing K cannot keep both images seen and image tokens seen constant. Loss is a lens, not a substitute for post-SFT selection, and it is a weak predictor of TextVQA. Compute limits rule out a from-scratch scaling-law study. PMI and n-gram entropy are diagnostics; high-order entropy has finite-sample bias. The backbone is Qwen3 pure AR only, so rankings may move under another family. Downstream tests are a thin slice of generation and VQA, with no region grounding or fine spatial reasoning. The 8B recipe also lags Liquid-7B on MJHQ, so "good enough as a testbed" is not "matched generation quality".