All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts
Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin, Zhineng Chen
cs.CV
2026-09-21
ScriptMoE routes each image to top-2 script experts, hits 82.06% on 10 scripts, and lifts CC-OCR F1 from 65.71% to 80.89% by swapping the recognizer.
English and Chinese scene text recognition are close to saturated. Once a deployed OCR stack leaves those two languages, it still has to read Japanese, Korean, Arabic, Hindi, Cyrillic, Thai, and Tibetan. Two camps dominate. Expert systems run language ID, then send each crop to a per-language recognizer. A wrong language decision is unrecoverable, and the zoo of models grows with every language. Vision-language models fold many scripts into one network. They are heavy, expensive at the edge, and still inaccurate on several scripts.
A cleaner target is one recognizer for the major writing systems. Hundreds of languages collapse into far fewer scripts. PP-OCRv5 MLT already covers 106 languages with about ten script-specific models. This paper consolidates the world into ten scripts (Latin, Cyrillic, Chinese, Japanese, Korean, Arabic, Hindi, Thai, Bangla, Tibetan) and 229 languages. The data hole remains: outside English Union14M and Chinese BCTR, large-scale real scene-text training sets barely exist.
They first synthesize the missing supervision. TextMuSS-10M, built on the UnionST engine, has 1 million images per script. Vocabularies follow PP-OCRv5 and standard character sets. Each script gets 100k to 1M words, plus concatenated phrases, random rare-character strings, and News Crawl sentences. Vertical text is 20% for Chinese, Japanese, and Korean, 5% otherwise. Arabic is rendered right-to-left and stored in logical order. Training mixes real English, real Chinese, a little real multilingual data from MLT2019, and the 10 million synthetic images.
TextMuSS-Bench reuses the MLT2019 test split for seven scripts and adds real Tibetan, Russian, and Thai images, 10,899 crops in total. Latin alone has 5,885; Chinese has 325.
The modeling bet is simple: a scene-text crop almost always contains one script, so routing is rarely ambiguous. ScriptMoE keeps a shared SVTRv2 visual encoder and replaces only the decoder FFN with sparse experts. Ten scripts are grouped by glyph family into four experts: alphabet (Latin plus Cyrillic), CJK, Arabic, and others (Hindi, Bangla, Tibetan, Thai). An image-level router pools visual tokens once and picks the top-2 experts; every output token of that image shares the route, which avoids token-level ping-pong. A shared expert stays on for digits, punctuation, and geometric distortion. A four-way script classifier sits on the same pooled features, with labels derived from Unicode ranges of the ground-truth string. Its loss weight is 0.1, a nudge rather than a hard assignment. The decoder has two layers, matching the SVTRv2-AR baseline, so the only architectural delta is the MoE FFN.
Every STR baseline was retrained on the same mixture for the same number of epochs. ScriptMoE reaches 82.06% mean word accuracy on TextMuSS-Bench, 1.31 points above SVTRv2-AR at 80.75%. Gains sit on the hard scripts: Arabic 78.09 vs 75.11, Thai 72.00 vs 69.60, Tibetan 88.76 vs 86.80. Latin drops 0.60 points (91.67 vs 92.27) and Hindi drops 0.51.
Against generalist systems the gap is much larger. PP-OCRv5 MLT averages 63.59%. Qwen3.5-9B scores 63.77% zero-shot. InternVL3.5-8B sits at 24.94% and DeepSeek-OCR2 at 8.61%; several models are near zero on Arabic, Bangla, and Tibetan.
On the CC-OCR multilingual end-to-end task, the PP-OCRv5 detector is frozen and only the recognizer changes.
| Method | Metric | Score |
| ScriptMoE | TextMuSS-Bench mean word acc. | 82.06% |
| SVTRv2-AR | same-data retrain | 80.75% |
| PP-OCRv5 Det + ScriptMoE | CC-OCR multilingual F1 | 80.89% |
| Qwen3.5-9B | CC-OCR zero-shot F1 | 80.73% |
| PP-OCRv5 MLT | CC-OCR F1 | 65.71% |
The model stores 45.85M parameters and activates 41.13M per image. On a V100 at batch 256, latency is 541 ms versus 396 ms for SVTRv2-AR and 1298 ms for MAERec. On saturated monolingual sets it still edges the dense baseline: 86.85% vs 85.93% on Chinese BCTR, 88.95% vs 88.67% on English Union14M-Benchmark.
Four experts are the sweet spot. Ten experts, one per script, fall to 81.32%; each expert sees too little data. Removing the shared expert costs 0.71 points on average and about 3.6 points on Arabic and Tibetan. Real-only training scores 0 on Russian, Thai, and Tibetan. Synth plus real lifts the seven MLT2019 scripts from 77.12% to 85.52%.
Deployed multilingual OCR today usually means a per-language model zoo or a 9B VLM. This paper offers a third option: a 46M recognizer whose end-to-end F1 matches a 9B-class VLM. Code lives in OpenOCR. For stacks already running a PP-OCR detector, swapping the recognizer is the cheap move.
The MoE itself is a modest +1.31 over a matched dense decoder. The larger jump is versus per-language experts and general VLMs. Synthetic data may be doing as much work as the architecture: without TextMuSS-10M, three low-resource scripts cannot be trained at all.
The authors list three. A domain gap remains between synthetic TextMuSS-10M and real scenes. Latin-Cyrillic homoglyphs are reduced, not solved: visually identical letters still get transcribed into the wrong script. End-to-end scores are still capped by the upstream PP-OCRv5 detector, especially on Latin, which is scored at word level.
The benchmark is unbalanced (5,885 Latin images, 325 Chinese), and the headline metric is an unweighted mean across ten scripts, so small scripts weigh as much as Latin. STR baselines were retrained on this data; VLMs were zero-shot. That comparison is fair as a deployment snapshot, not as a claim that specialist recognizers dominate VLMs after equal finetuning. Hindi and Latin regress versus the dense AR baseline. Inference is about 37% slower than SVTRv2-AR. Continual addition of new scripts is listed as future work.