Scaling Properties of Text Conditioning in Visual Generation
Zilong Chen, Chaorui Deng, Kunchang Li, Hongyi Yuan, Haoqi Fan
cs.CV
2026-08-01
In text-to-image, verbose prose captions saturate fast; only structured JSON prompts keep lowering diffusion loss. ByteDance's full system on Qwen-Image beats open and closed models on most benchmarks.
Text-to-image models learn from image-caption pairs. A language model learns from the text stream directly, but a text-to-image model learns text-conditioned generation only through the caption. That hides an overlooked asymmetry. Visual content a caption binds ambiguously reaches the model weakly, and content the caption omits does not reach it at all, even when it is plainly in the pixels. The image-grounded information in a caption can therefore cap what a generator can ever learn to recover from text.
The ByteDance Seed team treats text conditioning as a training variable in its own right and asks a direct question: does increasing this information improve generation, especially on prompts dense with objects, layouts, and relations where current systems struggle?
They start with a fixed-backbone reconstruction probe. From one reference image they verbalize the annotation into progressively longer natural-language (NL) captions and condition the same trained diffusion model on each to reconstruct. Quality saturates quickly, because these NL captions merely elaborate the same entities and relations rather than adding new ones. Verbosity alone does not help; visual variables must be made explicit.
So they introduce structured prompts (SPs), a typed JSON semantic representation with named fields for global scene context, per-element properties and geometry, and cross-element relations. Reconstruction rises steadily as schema coverage grows. To quantify caption informativeness they adapt two complementary metrics: GPG (grounded perplexity gain, how much revealing the image raises caption likelihood under a frozen VLM) and ED (effective detailness, precision and recall of caption attributes against image-grounded references). Sweeping formats and detail levels under a fixed recipe, converged diffusion loss is linear in GPG and power-law in ED, the scaling properties of text conditioning. Once calibrated, these can rank candidate caption conditions before a training run.
They name the format-side capability diffusability, raised through an image-to-SP pipeline that pairs a VLM with frozen pose, depth, and segmentation experts. At test time there is no paired image, so an LLM prompter must infer plausible details into the SP; this capability is promptability, raised through SFT, cold-start distillation, and verifier-gated RFT (the final stage distills an image-conditioned teacher on verified on-policy rollouts, OPSD), with a refine-render-judge agentic loop revising fields at inference.
The full system, on a Qwen-Image backbone, leads almost every open-weight metric and matches or beats closed systems on most. It reaches GenEval 0.94 (vs 0.91 for Qwen-Image with official prompt enhancement), GenEval2 geometric mean 72.5 (vs 52.8), DPG 90.71 (vs 87.20), WISE 0.89 (vs 0.83), and CoReBench 85.2 (vs 74.7), with the widest margins on composition- and reasoning-heavy benchmarks. Crucially, a matched-NL control retrains the same Qwen-Image backbone and prompter with a free-form caption interface; it does improve (GenEval2 GM to 56.2, CoReBench to 76.1) but stays well below the SP system, so the gain comes from the structured interface rather than a bigger backbone or more training. A prompter-training ablation shows SFT gives the largest single-stage structure gain (4.86 to 6.27), and the trained prompter in the inference loop reaches 54.7% GSB at eight rounds, above the strongest coding agent at 44.7%. Returns saturate quickly: the trained prompter passes in 2.31 rounds on average, and widening from four to eight rounds adds almost nothing.
This elevates something long dismissed as "caption quality" into a calibrated training variable for text-to-image. For trainers, the takeaway is immediately usable: structured annotation beats piling on words, and GPG and ED can predict training loss. The practical signal is that field-level revision beats long-horizon prompt-side reasoning; text-to-image does not need much of the latter.
This is a heavy system and engineering report. Evaluation leans on expensive offline judges (GPT-5.4, Gemini) that themselves change over time. The backbone is Qwen-Image, so some capability is inherited from the backbone and hard to isolate. The SP schema is bespoke and would need adaptation elsewhere. The "prose does not scale" claim holds under their controlled recipe and is untested under other data distributions. The agentic loop adds rendering cost.