OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?
Jiaxin Ge, Yiming Qin, Ji Xie, Haozhe Jiang, Xiaochuang Han, Junyi Zhang, Andrew Dai, Yinfei Yang, Jitendra Malik, Ranjay Krishna, Sewon Min, Haiwen Feng, Le Xue, Baifeng Shi, Trevor Darrell, XuDong Wang
cs.CV
2026-09-30
On BAGEL, I2I-then-I2T scales with I2I data while mixed training does not. Inpainting lifts 2D ordering +7.2 pp; Euclidean depth lifts metric 3D +3.8 pp vs I2T-only.
Unified omni models train visual generation and visual understanding in one network. Prior work has shown that understanding signals improve generation. The reverse has stayed muddy. Several papers conclude that generation helps understanding far less.
That claim sits awkwardly with the supervision itself. Pixel-level generation covers appearance, spatial layout, and geometry, which recognition, counting, spatial reasoning, and 3D perception all need. The failure mode may be the training recipe, not the generation objective. The study asks three questions: which curriculum lets generation help understanding, which generation tasks help which skills, and whether gradient alignment can flag useful pairings. All experiments run on BAGEL.
BAGEL-7B-MoT uses two Transformer experts that meet in shared attention. Text and ViT tokens go to the understanding expert; VAE latents go to the generation expert. The I2I objective is flow matching, learning a velocity field from noise to the target image, and that loss can backprop through attention into the understanding branch.
The controlled tests reuse Jigsaw and Zoom-In from VisGym. Same image, same visual problem: I2I emits the correctly ordered image, I2T emits the permutation as text. Among six curricula, I2I then I2T wins. Generation is trained first while shared parameters stay trainable, then understanding is finetuned. Freezing those shared weights during I2I, or mixing both losses from step one, gives weaker or unstable gains. Later experiments stick to I2I then I2T.
OmniTaskonomy puts both sides on one tree under Malik's 3Rs: Recognition, Reconstruction, Reorganization. Twenty-five understanding capabilities are induced bottom-up from BLINK, MMStar, MMT-Bench, CV-Bench, RealWorldQA, MMVP, and VStarBench. Nineteen I2I tasks are placed by the capability their target image directly supervises, each trained on 50k pairs from Taskonomy tiny, VisGym, COCO, or ConceptEdit-12M. Of 10,422 eval items, three VLM judges keep 9,444 by majority vote. Four people audited 250 items; among definite judgments, 97.4% matched the VLM label.
Transfer runs use about 50k I2I examples then 50k LLaVA-Instruct, against an I2T-only baseline, three seeds, exact paired permutation tests at p<0.05. Gradient alignment is cosine similarity of I2I and I2T gradients in a shared PCA subspace at the pretrained checkpoint. In the paired tasks, the strongest match sits in the understanding branch's early pre-attention RMSNorms.
As I2I data grows from 3k to 100k, I2I-then-I2T accuracy on the paired I2T task keeps rising. Mixed training does not show the same scaling. The substitution effect is largest when I2T data is scarce.
| Setup | Baseline | Result |
| Zoom-In: 100k I2I + 1k I2T | 10k I2T-only | comparable accuracy |
| Jigsaw: 100k I2I + 3k I2T | 10k I2T-only | comparable accuracy |
On a 30k I2I + 1k I2T checkpoint, Jigsaw I2T accuracy is 88.2% when the paired I2I output is correct and 64.3% when it is not.
Transfer across OmniTaskonomy is selective. Only 5 of 19 mapped capabilities improve significantly from at least one I2I source. Metric 3D relation and counting each pick up 12 sources; 2D ordering picks up 11. OCR, text recognition, and appearance understanding get no such lift.
| I2I source | Capability | vs I2T-only |
| Euclidean depth | metric 3D relation | +3.8 pp |
| Z-depth / surface normals | metric 3D relation | +3.6 / +3.4 pp |
| Jigsaw | 2D ordering | +6.8 pp |
| Inpainting | 2D ordering / counting | +7.2 / +1.5 pp |
| localization / object pointing | counting | +2.5 / +2.0 pp |
| 2.5D segmentation | category recognition | +1.2 pp |
The heatmap also has red cells. Across seven high-count capabilities, mean gradient alignment correlates with mean transfer at r=0.795; across 133 source-target pairs, r=0.529.
For omni-model training, the usable recipe is: train generation alone first, keep shared parameters unfrozen, then finetune understanding. Mixing from the start is a bad bet if the hope is that more I2I data will lift I2T.
Task choice matters more than adding a bit of generation. Depth and normals are the right sources for metric 3D; localization and pointing are the right sources for counting. Gradient alignment is a plausible pairing filter, but r=0.529 is only a moderate correlation, not an automatic curriculum search.
This is a controlled mechanism study, not a new SOTA table. Gains on BAGEL-7B with 50k-scale I2I sit mostly between 1 and 8 points, and only for some skills.
There is no standalone Limitations section. The bounds are still visible.
Only BAGEL's MoT is tested. Shared-attention gradient flow may not carry over to Janus-style split visual encoders or Transfusion-style AR-plus-diffusion. I2I correctness is scored by gemini-3.7-flash; the taxonomy leaves are VLM-labeled, with a human audit of 250 items. Negative transfer shows up on the heatmap, yet the paper spends little space on why those pairs hurt. Transfer-stage I2T data is generic LLaVA-Instruct, not matched to the target capability. Several leaves are tiny (part recognition 54, 3D shape 30, occlusion 32), and the main map drops capabilities under 100 examples. The gradient result is a correlation, not a causal claim.