DeCoRL: Decomposed Consistency Reinforcement Learning for Multi-Image Composition
Zhiqiang Wu, Shuang Sun, Jiale Zhang, Jing Li, Yibo Chen, Xu Tang, Yao Hu
NeurIPS
cs.CV, cs.AI
2026-10-29
DeCoRL splits multi-image consistency into per-object VLM scores, then runs diffusion RL. Six references go from 6.15 to 7.61; MICo-Bench goes from 37.58 to 54.03.
Multi-image composition (MICo) places people, clothes, and objects from several references into one frame under a text instruction, and it has to keep identity and appearance. Single-image editing already follows instructions reasonably well. From two references to six, the mapping turns many-to-one: a face, a color, or a material gets inherited from the wrong photo.
The usual training mix is a large multi-image set, often synthetic, plus supervised fine-tuning (SFT) that imitates a target image. The loss sits on the full output and never marks which reference each object came from. Named objects then drop out, and identity and attributes get mixed. Qwen-Image-Edit and FLUX.2 already accept multiple references. Closed systems such as Nano-Banana-Pro and Seedream 4.0 still rewrite identity and attributes once the reference count grows.
One consistency grade from a vision-language model over the whole set is unstable for the same reason. Cross-image attention is limited, and even GPT-4o is unreliable at that holistic call. A noisy reward leaves reinforcement learning without a signal it can trust.
The system is from Xiaohongshu and starts from Qwen-Image-Edit2511. Past three references the base model is unreliable, so SFT comes first. MICo-150K and UniPic 3.0 supply about 470K multi-image examples spanning one to six references. Nano-consistent-150K is mixed in so single-image editing does not drop away. All references share a pixel budget, 1024 squared in the paper's example, and one scale factor keeps aspect ratios while the visual tokens still fit. Flow matching trains the generator to predict latent velocity, not the clean image.
DeCoRL replaces one whole-image match with a match on every object the instruction names.
A vision-language model reads the instruction and the references, writes an open-vocabulary label for those objects only, and ignores background. Grounding DINO, a detector that boxes a text phrase, keeps the top-confidence box on the reference and crops it, then searches the generated image for the same label. A missing box scores 0, which pulls the average down when an object is dropped. When both crops exist, Qwen3-VL-32B fine-tuned on PaCO-RL scores the pair. Yes/no targets alone are easy to shortcut, so training requires a chain-of-thought explanation before the decision. Scoring stops at the first top token that is yes or no and applies a sigmoid to the logit gap, a value in [0, 1]. If neither token appears, the score is 0.
The multi-object reward is the mean of these scores, zeros included. The mean is intentionally crude, so the worst object can stay bad.
Diffusion NFT runs the update. For a group of samples under one instruction, rewards become weights in [0, 1]. High and low scores push the velocity prediction in opposite directions, and a penalty on the gap to a reference model's velocity limits drift. The decomposed reward does not depend on this optimizer. Only a rank-32 LoRA is trained; the base stays frozen. Crops miss global distortions, so a second penalty from Qwen3.5-122B-10A discourages unnatural images, warped structure, compositing artifacts, and implausible interactions. That term makes reward hacking harder.
SFT uses 32 H800 GPUs, batch size 128, 24,000 steps, and learning rate 2×10^-5. RL starts from that checkpoint on MICo-150K only, again on 32 H800s, with group size 16 and learning rate 3×10^-4.
DeCo-Bench is a new set: 40 real-photo cases at each count from 2 to 6 references, 200 in total. Higher is better.
| Method | Metric | Score |
| Qwen-Image-Edit2511 | DeCo-Bench overall | 6.22 |
| SFT | DeCo-Bench overall | 7.05 |
| DeCoRL | DeCo-Bench overall | 8.16 |
| Qwen-Image-Edit-MICo | DeCo-Bench overall | 5.17 |
| Qwen-Image-Edit-UniPic3.0 | DeCo-Bench overall | 6.02 |
| FLUX.2-dev | DeCo-Bench overall | 7.10 |
| Seedream 4.0 (2K) | DeCo-Bench overall | 7.76 |
| Nano-Banana-Pro | DeCo-Bench overall | 8.69 |
| Base model | 6 references | 3.64 |
| SFT | 6 references | 6.15 |
| DeCoRL | 6 references | 7.61 |
| SFT | MICo-Bench overall | 37.58 |
| DeCoRL | MICo-Bench overall | 54.03 |
| FLUX.2-dev | MICo-Bench overall | 49.55 |
| Nano-Banana-Pro | MICo-Bench overall | 51.76 |
The prose says SFT improves every bucket. Table 1 does not: two references fall from 8.41 to 8.34, and four fall from 7.09 to 6.80. DeCoRL brings those buckets back to 8.91 and 8.05. Six references move from 6.15 after SFT to 7.61; the base model is at 3.64 there, and at 4.98 with five references. Nano-Banana-Pro still leads DeCo-Bench at 8.69, ahead of 8.16.
On public MICo-Bench the same three checkpoints score 22.90, 37.58, and 54.03. The last number is above Nano-Banana-Pro at 51.76 and FLUX.2-dev at 49.55. Human Centric moves from 30.49 to 56.21, De&Re from 45.50 to 64.42, Object Centric from 39.60 to 53.47, and HOI only from 32.41 to 42.49.
Against SFT, subject resemblance goes from 7.01 to 7.64, naturalness from 6.54 to 8.01, and the artifact score (higher means fewer artifacts) from 5.76 to 6.64. Prompt following slips from 8.76 to 8.69.
An untrained vision-language reward reaches only 7.21 overall, and at four references it falls from 6.80 to 6.42. The CoT scorer reaches 8.16. During training the raw reward stays mostly under 0.5, while the trained one stays above 0.6 and climbs. A prompt is banned once its samples already earn high, similar rewards, a sign the signal is learned. That ban count stays at 0 for the raw model and rises for the trained scorer.
The lift over SFT sits at five and six references, and on people. At two references SFT itself slips from 8.41 to 8.34, and DeCoRL is what returns the score to 8.91.
What transfers is the split. When a holistic grade is unreliable, mark the objects named in the instruction and score them with a single-object model. The detector, the scorer, and the optimizer are written as replaceable parts. A second backbone is not tested.
Both stages use 32 H800 GPUs, and the reward path also runs a 32B scorer plus a 122B-class general model. The deployed editor keeps the LoRA. This is not a cheap post-training pass.
On DeCo-Bench, 8.16 still trails Nano-Banana-Pro at 8.69. On MICo-Bench, 54.03 leads its 51.76. Correspondences that SFT leaves implicit get better. Winning every board does not follow.
The main reward only sees objects a detector can box. Relations, interactions, lighting, and anything that cannot be localized get no score. Stronger matching, aggregation that protects the low-scoring object, and any test past six references are left open. There is no run beyond six.
A mean hides a miss. Four strong objects and one failure still look fine on average. That average does not guarantee balanced gains. Top-1 boxes add another failure: a repeated label or a shifted box rewards the wrong region, or marks a generated object as missing.
DeCo-Bench has 40 cases per count and was built for this paper. Who scores it, what the maximum is, and whether that judge shares the training reward's taste for local crops are not reported. A gap of about half a point versus Nano-Banana-Pro is not yet a stable ranking.
Naturalness moving from 6.54 to 8.01, and the artifact score from 5.76 to 6.64, sit next to the auxiliary model that punishes warped and composited outputs. No run removes that term, so the two contributions stay mixed. Prompt following lands about 0.07 below SFT, a small move the other way.
The scorer is trained on PaCO-RL crops and applied to multi-image crops with no separate measure of that shift.