Single-Step Error Correction for Image Understanding and Generation

google · hf · 2026-07-17

Enabling Image Understanding and Generation to Self-Correct in the Same Step

This work proposes SC-CMJP (Self-Correcting Coupled Markov Jump Processes) and the corresponding sampler CO₂Jump. The core idea: image generation/understanding shouldn't be disjoint tracks, but a coupled process of "mutual correction". The authors argue that existing masked diffusion sampling methods—whether interleaved decoding or parallel updates—struggle to let both modalities check each other in real-time within the same step; and since models cannot re-mask, cross-modal conflicts are hard to fix once they occur.

Method

Uses the transition rate of one modality to depend on the other's confidence
Introduces remasking jump: revokes made commitments when cross-modal evidence opposes the current decision
Based on this mechanism, proposes a training-free single-pass sampler, CO₂Jump

Data & Evaluation

The authors also constructed and will release three large-scale joint multimodal generation corpora:

JEdit-1M
JMaze-200K
JNono-200K

The accompanying benchmark covers both in-distribution and out-of-distribution scenarios. Experiments show that CO₂Jump achieves the best performance in image understanding/editing and visual reasoning (like maze and Sudoku-style tasks), with performance improving as sampling steps increase, proving that cross-modal coupling benefits continuously accumulate along the trajectory.

Original post →

Single-Step Error Correction for Image Understanding and Generation

More from Multimodal