Real In, Real Out: What If We Only Use Real Data for Scene Text Editing?
Xingsong Ye, Yongkun Du, Jiaxin Zhang, Chong Sun, Zhineng Chen
NeurIPS 2026
cs.CV
2026-10-20
TextRIRO trains scene-text editing on real images, via column-shuffle and crop self-pairs. Variable-length accuracy is 47.6% versus ~12%; the 3M set adds 1.91 recognition points.
Scene text editing (STE) replaces the words in a photo and keeps color, texture, illumination, stroke weight, and perspective. Real photos almost never offer a pair with the same background and different words, so training has long relied on rendered pairs.
Two-stage models such as SRNet split foreground from background and fuse them again; source glyphs often remain. Implicit disentanglement, as in TextCtrl, generates from pretrained features and tends to drop or misspell characters. MOSTEL and RS-STE bring in real images by erasing then writing back, or by reverse self-supervision, but synthetic samples stay the bulk of training. Outputs keep a synthetic look, and the style often slips. Inpainting editors such as AnyText and TextSSR fill a masked text region from context. Without a nearby style match, the strokes do not return.
Style lives mostly in low-level pixels. Meaning lives in character order. Break the order, keep the pixels, and one real image can supervise itself.
TextRIRO (Real In, Real Out) is a conditional diffusion model trained only on real scene-text images. The groups are Fudan University's Institute of Trustworthy Embodied AI, the Shanghai Key Laboratory of Multimodal Embodied AI, and WeChat Vision at Tencent. The paper is in NeurIPS 2026.
A labeled photo is split along its long side into L equal columns, one per character. With some probability those columns are shuffled, then a contiguous run of at most L columns is cropped out as a pseudo-source. That is a self-pair: the same real image supplies the style and the reconstruction target. The network rebuilds the original from the pseudo-source and the original label. At test time the pseudo-source is the image to edit, and the label is the new string.
Column shuffle breaks left-to-right order while color and stroke stay inside each column, which fits straight text. Tilt and curvature sit in contiguous patches, so a crop keeps a local piece. Masking wipes style out with the content. Block reordering shifts the training distribution. With no perturbation, ScenePair accuracy falls to 4.37% and the model copies its input.
Style features come from a ViT and a spatial pyramid, injected into the U-Net through a ControlNet branch, the side path that feeds spatial conditions into the diffusion net. The text encoder is pretrained with an autoregressive objective on real scene text, the same job as scene text recognition (STR): read the string. It stays frozen with the VAE. The style encoder and U-Net take the noise-prediction loss. Inputs are 64×256 so a square resize does not squash glyphs, and the VAE is SD 2.1. An OminiControl DiT on the same pairs scores 52.89% accuracy (FID 39.65), against 85.23% (FID 41.53) for ControlNet. Token-level conditions do not hold strokes.
Edits still fail often enough that the images cannot train a recognizer alone. TextRIRO-3M starts from U14M-Filter, edits each image to a different equal-length string, keeps a sample only when a separate real-data STR reads the target exactly, and concatenates it with the original. Sequence length doubles. The set has 3 million images.
On ScenePair, scored by the official TRBA recognizer, lower FID means closer to real-image statistics and higher NED means the recognized string is closer to the target. TextRIRO has the lowest FID in the table and sits 3.21 points under real images.
| Method | ACC | NED | FID |
| TextRIRO | 85.23% | 0.964 | 41.53 |
| RS-STE | 91.80% | 0.981 | 49.59 |
| TextCtrl | 84.67% | 0.936 | 43.78 |
| TextSSR | 79.69% | 0.924 | 45.37 |
| Flux.1 Kontext | 12.81% | 0.460 | 41.59 |
| Qwen-Image-Edit | 53.16% | 0.825 | 49.91 |
| Real images | 88.44% | 0.974 | 0 |
RS-STE beats the real-image ceiling on accuracy and NED, with FID 8.06 worse than TextRIRO. That pattern fits recognizer-friendly synthetic ink. Flux.1 Kontext posts FID 41.59, 0.06 from TextRIRO, at 12.81% accuracy. SVTRv2 and PPOCR-v6 show the same split: TextRIRO at 85.86% and 86.25%, ceilings at 88.67% and 89.14%, RS-STE still above the ceiling.
RIROCommon uses SVTRv2 and scores both an equal-length target and a variable-length target. Mean accuracy is 79.31% / 47.64% for TextRIRO, 70.67% / 11.98% for TextCtrl, and 60.24% / 11.73% for RS-STE. Equal-length accuracy is 19.07 points above RS-STE. The variable-length gap is wider. On curved CUTE, equal-length accuracy is 64.87%.
Training a recognizer on the edited images tests whether they are usable data. Mean accuracy is 86.67% for TextRIRO, 76.89% for RS-STE, 75.99% for TextCtrl, and 96.77% for the same volume of real data: 9.78 points over the runner-up, 11.69 points on irregular sets (82.56% vs 70.87%), and 10.10 points short of real data.
In the ScenePair ablation, a text encoder with no pretraining scores 9.38%. STR pretraining, 64×256 inputs, and SD 2.1 score 82.42% (FID 43.59). Shuffle only is 45.78%, crop only 80.08%, mask 13.13%, block reorder 79.36%. No perturbation scores 4.37% with FID 33.28: the picture looks real and the string is wrong. On straight IC13 the combo leads at 80.16%. On curved CUTE, crop only leads at 60.93% versus 58.42%. On multi-oriented SVTP, crop only leads at 69.76% versus 68.84%. The main table's 85.23% and this ablation row's 82.42% do not match, and the paper does not say where the gap comes from.
MAERec trained only on 3.22M U14M-Filter images scores 86.09% on Union14M and 86.89% on multi-word. Mixing in 1M concatenations reaches 87.98% (+1.89) and 89.08% (+2.19). At 3M the scores are 88.00% (+1.91) and 90.17% (+3.28). Common benchmarks average 96.98% (+0.21), and IIIT drops 0.10. MJ, ST, MOSTEL, TextCtrl, and RS-STE all land lower. The nearest, TextSSR, is 87.85%, 0.13 behind. Real-Concat, two unrelated real images joined together, scores 87.33%. On SVTRv2-S, the 3M mix adds 1.34 points on Union14M and 2.30 on multi-word.
Equal-length accuracy moves from TextCtrl at 70.67% to 79.31%, a steady step. Variable-length accuracy moves from about 12% to 47.64%. After the length changes, a much larger share of edits still spell the target. The crop used in training is a plausible reason: the model has seen fragments of different lengths.
Common recognition sets are already full. The lift is on Union14M multi-word and artistic text, not on IIIT near 99%. Style-matched joins beat unrelated real photos, 87.98% versus 87.33% at the 1M mix. The backbone can be swapped. Synthetic triplets are no longer the main training set.
Multilingual real scene text is scarce, so the experiments are mostly English. Low-resource multilingual editing is not attempted. Wrong edits leak into the pool. Used alone as recognition data, edited images still trail an equal amount of real data by 10.10 points.
On ScenePair accuracy and NED, RS-STE is higher, and it beats real images on all three recognizers. Taking that one accuracy number as a win does not hold. At 47.64% variable-length accuracy, failures are still the majority. CUTE equal-length accuracy is 64.87%, so curved text is harder. On curved and multi-oriented subsets, crop alone is better. The combo helps straight text.
The +1.91 and +3.28 points belong to MAERec at the 3M mix, on Union14M overall and on multi-word. Common benchmarks barely move, and IIIT falls. Real-Concat already goes from 86.09% to 87.33%, and TextRIRO reaches 87.98%. TextSSR is only 0.13 behind. Lengthening the string does much of the work, and the strongest editing-based baseline sits close.
There is no human preference study. A 0.06 FID gap can hide a seventy-point accuracy gap, as Flux shows. General editors saw only the fixed prompt "Edit the text to 'Ttgt'", so their scores may be low. Column cuts assume text runs along the long side and that the labeled length is right. Vertical text, touching glyphs, and bad labels slice the wrong columns. The shuffle probability is never given a number in the main text.