Learning Native Reflection in Unified Models with Interleaved Reinforcement Learning
Yijia Fan, Ziqi Huang, Zhongang Cai, Yan Li, Zimo Wen, Wanqi Yin, Haiwen Diao, Ziwei Liu
cs.CV, cs.AI
2026-09-29
Trajectory-level RL on BAGEL updates reflection text and image rendering with one shared advantage: GenEval +12 over SFT, repair rate 21% to 65%, transfers to 3 unseen benchmarks.
Unified multimodal models put image understanding and image generation in the same network, so the model that renders an image can also look at it. In principle it can repair its own generations: render, inspect, diagnose, revise, look again. The training signal is the hard part. SFT on distilled reflection trajectories gives a cold start, but imitation teaches well-formed critiques, not repairs that land. Existing RL work optimizes a single render (T2I-R1, UniRL) and never revises it. Pipelines with an external critic (GenArtist, Idea2Img) keep the critic online at inference, and the two models are never optimized against each other. What is missing is outcome-driven RL over the unified model's own multi-round reflection. On the text side, SCoRe already showed that multi-turn self-correction needs online RL; this paper carries the lesson to pixels.
The backbone is BAGEL, whose understanding and generation experts share attention, so one advantage can push both heads. Two stages:
The design choices have concrete reasons. Shared roots make the group-relative advantage compare reflection strategies rather than lucky first draws. The advantage sits at the trajectory level, not per round: per-round branching needs K^N rollouts (16^3 = 4,096 for three rounds), and a PPO-style critic would require a value model over multi-round image-text states. The trajectory level keeps GRPO's K-sample cost while the per-round progress terms stay inside the reward. The reward is graded: under binary GenEval scoring, partial improvements yield zero delta in five of six families, so partial credit is granted only in the failing region, from the verifier's own detector outputs, and never crosses the pass line. The verifier runs during training only; inference is the unified model alone.
| Setting | GenEval | WISE | OneIG | CompBench |
| BAGEL-Base | 0.71 | 0.55 | 0.80 | 0.49 |
| Reflection SFT | 0.72 | 0.63 | 0.79 | 0.50 |
| UMM-Reflection | 0.84 | 0.74 | 0.83 | 0.55 |
The understanding head and the generation head of one unified model can be trained jointly under a single outcome reward, which separate critic-plus-renderer pipelines cannot do. The gains are not bought with inference compute: Best-of-4 at the same image budget reaches 80 against 84 here, with no verifier and no external model at deployment. A 65% repair rate means three revision attempts recover most failed generations, with the largest wins on position and counting errors. Training cost is contained: 1,000 updates, about 33 hours on 16 H100s, from a 3,000-prompt pool.
The RL prompt pool is drawn from GenEval's six families (the Flow-GRPO training set), so the +12 on GenEval has a same-distribution ingredient; transfer gains are smaller (OneIG +3.48) and some CompBench categories drop relative to Base (2D spatial -1.3, color -5.0). The damage rate on initially correct images is 8.8%, above SFT's 6.5%. Evaluation runs at 512x512 rather than BAGEL's native 1024; the paper's own explanation is that whole-trajectory sampling would not be tractable at higher resolution. The loop is capped at three repairs and driven by an external controller, since BAGEL does not natively support interleaved text-image generation in one forward pass. The graded reward is assembled from the GenEval verifier's detector outputs, so domains without a verifiable scorer (aesthetics, open-ended generation) cannot reuse it directly.