Text-to-Image Generative Modeling Without Language Pretraining
Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas, Ehsan Adeli, Jiajun Wu, Juan Carlos Niebles, Li Fei-Fei, Michael Poli
cs.LG, cs.CV
2026-10-20
Halcyon-1.6B, trained from scratch on 100M pairs, hits 35.3 gFD and beats JiT-T2I-XXL/8++ (41.0) with 275× less total pretraining compute.
Text-to-image models usually condition an image generator on a language model trained on web-scale text. Late fusion keeps that text encoder separate. Early fusion puts text and image tokens in one network, and still spends most of the run on text-only data. In representative systems that stage is more than 80% of pretraining FLOPs, and it is often left out of the number people quote for training a T2I model. Whether it actually helps generation had not been measured by turning it on and off on one dataset.
Halcyon, from Stanford and Radical Numerics, trains the text tokenizer, the image tokenizer, and the generator from scratch, with captions as the only language signal.
Halcyon is a decoder-only Transformer in the Qwen3 layout. Text and image tokens share one vocabulary and are trained with next-token prediction. A 256×256 image passes through a discrete tokenizer trained from scratch, following the Cosmos encoder–decoder and finite scalar quantization (FSQ), which bins features into a few scalar levels. Spatial downsampling is 8×, the codebook has 64,000 entries, and the 32×32 grid is 1,024 tokens. The text tokenizer is a byte-level BPE with 16,384 tokens, fit on GPIC captions plus OpenWebText. The generator never sees OpenWebText. With 256 special tokens the joint vocabulary is 80,640, and the embedding table is tied to the output head.
T2I sequences put the caption first and then predict image tokens. I2T reverses the order. Classifier-free guidance (CFG) combines conditional and unconditional logits at sampling time. The unconditional branch comes from dropping the caption on 10% of T2I rows. The loss is weighted cross-entropy, with image tokens at weight 0.5 so 1,024 visual targets do not swamp the caption.
Every run is one epoch on GPIC, about 100 million pairs, at 256×256, with global batch 1,024. The five sizes are 0.1B, 0.25B, 0.52B, 1.6B, and 4B, at sequence length 1,280. LlamaGen (autoregressive, with FLAN-T5) and JiT-T2I (pixel-space flow matching, regressing a velocity field on pixels, with frozen Qwen3-1.7B) are retrained on GPIC from the official code and then moved to the same 1,024-token image grid. Data and sequence length are matched.
The lead metric is gFD on GPIC Test-50K, the Fréchet distance between features of generated images and real ones. Lower is better.
| Method | gFD ↓ | GenEval ↑ | Total pretraining FLOPs |
| LlamaGen-XL/8 | 63.3 | 30.5 | 9.68×10²¹ |
| JiT-T2I-XXL/8++ | 41.0 | 43.2 | 3.68×10²³ |
| Halcyon-0.52B | 48.0 | 44.9 | 5.33×10²⁰ |
| Halcyon-1.6B | 35.3 | 53.5 | 1.34×10²¹ |
| Halcyon-4B | 28.4 | 56.4 | 3.08×10²¹ |
JiT reaches 41.0 only after LPIPS and a patch-size cut from 16 to 8. Before either change it sits at 67.9. The table uses that stronger JiT. Halcyon-0.52B beats LlamaGen-XL/8 with about 18.2× less total pretraining compute, and still trails JiT on gFD, 48.0 against 41.0. GenEval is 44.9 against 43.2. DPG-Bench is 76.84 against 76.85. At 1.6B, Halcyon passes JiT: gFD 35.3 against 41.0, GenEval 53.5 against 43.2, DPG-Bench 77.9 against 76.9, at about 275× less total pretraining compute. At 4B, gFD is 28.4 and GenEval is 56.4. Position is 51.0, against JiT at 27.3. DPG-Bench stays at 77.9, tied with 1.6B. The guidance scale that minimizes validation gFD falls from 7.5 at 0.1B to 4.5 at 4B.
Replacing FLAN-T5-XL with FLAN-T5-Base cuts the language encoder from 1.22B to 0.11B and language-pretraining compute by about 13.5×. LlamaGen gFD only moves from 63.3 to 66.7. On Halcyon-0.52B, spending the first 10%, 20%, or 50% of a fixed budget on text-only data moves gFD from 48.0 to 51.6, 52.8, and 65.1. A language stage plus a full T2I epoch, 1.5× the compute, lands at 49.2. Continual pretraining from Qwen3-0.6B, already trained on 36T text tokens, reaches 47.8. The same architecture from random initialization, with a larger vocabulary (216k versus Halcyon's 80,640), reaches 46.6. Language initialization is not better. Dropping the peak learning rate from 7.5×10⁻⁴ to 1×10⁻⁴, so those language weights are not overwritten quickly, makes continual pretraining worse: gFD 63.6.
Replacing the dropped 10% of captions with I2T keeps conditional and unconditional image-token counts fixed. At 0.52B, caption NLL falls from 18.4 to 1.1, gFD from 48.0 to 46.5, and GenEval from 44.9 to 44.6. At 4B, Qwen3-VL-Reranker-8B scores generated captions at 0.559, against 0.891 for GPIC ground-truth captions. Treating I2T as a separate task that consumes budget is different. At 0.52B, a 50% mix raises gFD to 76.0. I2T first, then T2I, reaches 54.2. With the same token counts, the mix drives caption NLL to 1.0, while the staged run is still at 20.4.
Qwen3-1.7B accounts for 99.8% of pretraining FLOPs in JiT-T2I-XXL/8++. If the product is a T2I model, that budget trained a general language model. Halcyon's language signal is the captions, about 4 billion tokens.
A team that already has an LLM does not save 275×. Language pretraining is a sunk cost. The new bill is the image generator, and Halcyon costs more there: JiT's generator is 6.76×10²⁰ FLOPs, Halcyon-1.6B's generator plus tokenizer about 1.1×10²¹. The GPIC evidence supports a narrower claim. A T2I stack built from scratch does not need an internet-scale text stage first.
Captioning is free only when I2T replaces caption dropout. Cutting extra budget for it makes generation worse.
The 275× figure counts language-model pretraining in the total. On vision FLOPs, Halcyon is higher.
Resolution stops at 256, and the main benchmark is GPIC, the training distribution. The paper says other datasets and higher resolutions are still open. Mixture and schedule ablations use only the 0.52B model. The initialization comparison uses Qwen3-0.6B. The discrete tokenizer still misses small text and fine glyphs, and at 256×256 none of the models reliably render legible text.
Swapping in Halcyon's tokenizer, which also lengthens LlamaGen's image sequence from 256 to 1,024 tokens, drops gFD from 147.0 to 63.3. The image representation does a large share of the work. The headline table uses baselines that were also moved to 1,024 tokens, so it is not a win over the original LlamaGen at 147 gFD. Autoregressive token prediction and flow matching are not separated. Resampling one checkpoint moves gFD by a standard deviation of at most 0.14. Training-seed variance is reported only at 0.1B, where that standard deviation is 5.5. Larger models have no such error bar.