What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models
Tengju Ru, Weitong Lian, Zecong Tang, Zhiheng Liu, Lingyi Meng, Haoran Li, Kaixuan Wang, Bohao Li, Xihui Liu
cs.CV
2026-10-14
GAFU compresses native generation features into 256 G tokens for question-time retrieval. On three UMMs it beats same-data SFT by 3.56-5.98 points and wins all 15 settings.
Unified multimodal models (UMMs) put image understanding and image generation in one network, but the visual front ends are usually separate. Understanding uses a language-aligned tower such as CLIP or SigLIP and compresses the picture into a short run of semantic tokens. Attributes, small objects, and spatial relations are thinned out before the language model sees them. Generation uses a VAE or VQ-VAE because it has to paint the image back, so layout, position, and local detail stay denser.
Attempts to let generation help understanding take two routes. One adds a generative loss that reshapes features or hidden states. It never reads the native generation features, and the two pathways want different representations, so the extra loss can pull them into conflict. The other paints an intermediate image at inference and then tries to understand that image. That adds a full generation pass and copies generation errors forward. Some methods also re-encode the painting into the understanding space, which brings the original bottleneck back.
Fine-grained understanding fails when the evidence never reaches the understanding pathway. The generation pathway often still holds it.
The measurement comes before the training recipe. Six UMMs are compared against themselves: BAGEL, Janus-Pro, Lance, Show-o2-7B, BLIP3-o-4B, and DreamLLM-7B. The probe set is 2,084 VisMin pairs with one minimal edit, covering object, attribute, count, and relation. Each semantic edit has a control in the same region with matched linear-RGB root-mean-square change, so a reaction to pixel noise does not count as semantic sensitivity.
In all 24 model-domain cells, control-corrected sensitivity is higher on the generation side. The gap is smallest for count and larger for attribute, object, and relation. It stays positive when the long edge is 128, 256, or 512. A strong response means the local reaction to the semantic edit is more than twice the control. Samples with a strong response only on the generation side average 28.18%; only on the understanding side, 2.48%. The net gap is 25.70 points, and all 24 cells point the same way. Shuffling the image on a 64 by 64 block grid before encoding shrinks the sensitivity gap in every cell. The extra signal depends on the original spatial layout.
GAFU is post-training. The native encoders stay frozen. Generation features are projected to 1024 dimensions, joined with 256 learnable queries, and run through a 6-layer Transformer with 8 heads and 2D RoPE. The queries pool that information into 256 G tokens and are projected to the understanding width: 3584, 4096, and 2048 on BAGEL, Janus-Pro, and Lance. A decoder must rebuild the generation features from G, then reconstruct the image through the frozen generator. This stage uses 1 million Cambrian images.
Three losses follow the measurements. Reconstruction is image L1 plus 0.5 L2 plus 0.5 LPIPS, a perceptual distance. Spatial binding skips the decoder: each G token uses a linear head to predict the generation target in the same grid cell. Relation alignment does not match raw feature values. G and the understanding features U are pooled to 4 by 4, 8 by 8, and 16 by 16, plus one node per image, and the loss matches pairwise cosine relations among those nodes.
After every understanding-pathway layer, cross-attention reads G. Current text hidden states are the queries, G tokens are keys and values, and the output is added back onto the text stream. Positional encoding matches the native understanding tokens, and the output projection starts at zero, so step zero leaves the original model unchanged. The positive branch sees G for the current image. The negative branch swaps in G from another image and keeps the question and the understanding features U. The loss is answer cross-entropy, a pairwise DPO objective in cross-entropy space, and a forward KL that holds the negative branch near a frozen model conditioned only on U. Positive and negative DPO gradients are weighted 4 to 1, the negative gradient norm is capped at 0.25 times the positive norm, and the KL weight falls from 1 to 0 across 5,000 steps. That blocks a fake preference gap created by wrecking the shuffled branch. Early in training, U is fully masked with probability up to 0.8, so the model has to use G; the mask is later removed. The compressor is then unfrozen and trained with understanding-pathway LoRA, rank and alpha both 16. Downstream questions match the SFT run: 1,344,016 non-text FineVision samples with relevance at least 3, 7,680 VSR samples, and 19,654 PathVQA samples.
The runs are BAGEL 14B, Janus-Pro 7B, and Lance 3B, against same-data SFT and the untouched base. VStar-Bench, OmniSpatial, and MMVP pick the candidate answer with the lowest average token cross-entropy. That is not free generation. VSR-Bench uses greedy true/false generation. PathVQA uses greedy decoding and normalized exact match.
| Model | Setup | VStar | OmniSpatial | MMVP | PathVQA | VSR | Avg |
| BAGEL 14B | Base | 36.65 | 30.79 | 71.67 | 39.27 | 69.98 | 49.67 |
| BAGEL 14B | SFT | 39.27 | 31.90 | 74.00 | 51.42 | 71.39 | 53.60 |
| BAGEL 14B | GAFU | 47.12 | 34.51 | 79.67 | 59.07 | 77.54 | 59.58 |
| Janus-Pro 7B | Base | 31.94 | 25.64 | 54.67 | 16.91 | 55.72 | 36.97 |
| Janus-Pro 7B | SFT | 33.51 | 27.14 | 54.67 | 51.75 | 57.54 | 44.92 |
| Janus-Pro 7B | GAFU | 38.74 | 31.57 | 59.67 | 52.87 | 59.54 | 48.48 |
| Lance 3B | Base | 35.60 | 23.87 | 50.33 | 11.43 | 52.07 | 34.66 |
| Lance 3B | SFT | 36.13 | 24.72 | 50.00 | 43.04 | 53.26 | 41.43 |
| Lance 3B | GAFU | 42.41 | 29.16 | 53.67 | 46.91 | 54.35 | 45.30 |
GAFU beats same-data SFT in all 15 cells. The five-benchmark average rises by 3.56 points on Janus-Pro, 3.87 on Lance, and 5.98 on BAGEL. The largest relative gains reported are 20.00% on VStar (BAGEL, 39.27 to 47.12), 17.94% on OmniSpatial (Lance, 24.72 to 29.16), and 8.62% on VSR (BAGEL, 71.39 to 77.54). Absolute VStar gains are +7.85, +5.23, and +6.28. On MMVP, SFT is flat for Janus-Pro and drops Lance from 50.33 to 50.00; GAFU reaches 59.67 and 53.67. On the three general benchmarks, every GAFU-over-SFT gap is larger than the SFT-over-base gap. PathVQA is different: SFT already moves the base a long way, and GAFU adds another 1.12 to 7.65 points.
Ablations are on BAGEL only. Dropping spatial binding takes OmniSpatial from 34.51 to 33.14. Dropping relation alignment takes VStar from 47.12 to 45.55. Keeping both beats either one by 4.67 to 5.34 on MMVP. Concatenating G with understanding tokens trails by 1.96 to 6.28. Querying G with understanding-side visual tokens drops OmniSpatial to 22.37, under the base score of 30.79. Text hidden states are the better query. Ten injection layers sit within 0 to 1.67 of all 27. DPO without KL gains 1.05 on VStar and 8.00 on MMVP, then drops OmniSpatial from 33.33 to 22.24. Adding KL brings the three scores back up by 3.14 to 12.27.
The gain comes from wiring in generation features the model already computed, on the same question-answer data. Training touches the compressor, the injection layers, and understanding LoRA. It holds from Lance 3B to BAGEL 14B. Inference does not need to paint an intermediate image.
MMVP stays flat or gets worse under SFT alone, then moves once G is attached. The bottleneck sits closer to the visual features. The average gain is about four points, and the smallest cell is about one. Strip out KL and the spatial score collapses, so the loss recipe has to stay whole.
The paper states three limits. GAFU targets mainstream dual-pathway encoders, and a different interface needs a new adapter. Gains track the quality of the native generation features and vary by model and task. Extracting, projecting, and injecting G costs extra compute and memory versus the understanding pathway alone, and no latency or memory figure is reported.
Sensitivity is measured on six models; GAFU is trained on three. VStar has 191 items, so BAGEL's +7.85 is about 15 extra correct answers, with no multi-seed variance. The three headline benchmarks score candidate cross-entropy, and free-generation rankings can differ. Generative supervision and inference-time intermediate images are argued against on cost, without a head-to-head number on these benchmarks. Count is where the generation-side advantage is weakest, and the five benchmarks are not split by attribute, object, count, and relation. Stage one also uses 1 million Cambrian images, so the images used to learn G are not the images the SFT baseline sees, even though the downstream questions match. PathVQA is a research score. The paper says it must not support clinical decisions.