CMU Open-Sources T-DIG: Text-Conditioned Diversity for Image Generation

Researchers from CMU and Northeastern have released T-DIG (Text-Conditioned Diversity for Image Generation) fully open-source, tackling a longstanding pain point of text-to-image models: diversity. In a Halloween demo, the team asked an image model for "people in Halloween costumes" and sampled i.i.d. 8 times—often 4–8 of the results were the same witch, while prior unconditional diversity methods typically changed faces but not outfits.

Confirmed

• Authors Simran Khanuja (CMU), with collaborators Rohit Gandikota, Gaurav Parmar, and Graham Neubig, released the paper, code, demo, and a diversity dataset; the work builds on their TPIPS evaluation metric released in parallel.
• Root cause: image models are optimized for high alignment between a single image and a prompt, at the cost of diversity across batch generations. Prior methods only push images apart in embedding space, which may not match the dimensions users actually want varied. T-DIG's key idea: let diversity follow text instructions.
• Data construction: roughly 20,000 prompts were filtered from HPD V2 (logs of human-written prompts for Stable Diffusion); Gemini 3.8 Flash was then used, seeded with human-written "axes of variation," to synthesize new variation axes, which are released as a dataset.
• Training mechanism: an LLM policy generates diverse prompts from a given base prompt, trained on reward/penalty signals from generated images—➕ diversity along requested variation axes (TPIPS), ➖ alignment with the base prompt (VQA Score) and HPSv3 as penalties.
• The team showcased a classic reward hacking case: trained with diversity rewards alone, the LLM policy drifted far from the base prompt, producing blank images or piling on extraneous details across the gallery—showing that rewards and penalties must be combined.
• Effect demo: specifying variation axes such as "food-pun costumes" yields different outfits like pumpkins and pizza, rather than the same witch over and over.

Why it matters

T-DIG turns "diversity" from an unconditional embedding-space push into a text-controllable dimension, directly addressing users' real need to "get 8 different images in one go." The disclosed reward hacking cases also offer useful lessons for related reward-training research, and the full open-source release lowers the barrier to reproduction and follow-up work.

2026-10-28 ~ 2026-10-28 · 7 related posts

• T-DIG demo: ask for Halloween costumes, get food puns and ML researcher outfits instead of 8 witches — simi_97k · 2026-10-28
• Why image models give you 8 identical witches: the alignment-diversity tradeoff, explained — simi_97k · 2026-10-28
• [source] CMU researchers release T-DIG, an open framework for text-conditioned diversity in image generation — simi_97k · 2026-10-28
• T-DIG releases first text-conditioned diversity dataset with 20k filtered prompts — simi_97k · 2026-10-28
• [source] T-DIG trains an LLM prompt policy with TPIPS reward, VQA and HPSv3 penalties — simi_97k · 2026-10-28
• Reward hacking in T-DIG: diversity-only reward yields blank images and off-prompt sets — simi_97k · 2026-10-28
• [source] T-DIG ships full release: paper, code, demo, and first text-conditioned image diversity dataset — simi_97k · 2026-10-28