Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu, Jian Hu, Haihan Gao, Heting Gao, Xin Li, Yinsong Liu, Deqian Jiang, Di Yin, Haoyu Cao, Xing Sun, Bing Liu, Yanqiu Qu, Yunfei Wu, Kun Yin, Zhixiang Wei, Yongxiang Hua, Qianyu Li, Xiaoxing Hu, Peiyuan Zhang, Lijiang Li, Dong Xu, Zhenchuan Yang, Yiliang Jiang
cs.CV
2026-10-29
One open 5B checkpoint, one shared vision-audio encoder, one schema: ten parsing tasks, OmniDocBench SOTA at 96.96, best open-weight score on OmniParsingBench (75.08).
Agents increasingly act on documents, images, audio, and video rather than plain text. Feeding raw pixels and waveforms to the agent's own model is expensive and hard to audit: long visual and acoustic token sequences get re-encoded on every pass, and an answer cannot point to the region or interval that supports it. Parsing attacks this at the source: convert each input once into a structured text record whose entries carry bounding boxes or timestamps, and any LLM, text-only ones included, can read the record and trace claims back.
The landscape is fragmented. Document parsers are strong on standard pages, but chemical structures and sheet music still go to dedicated models, and audio/video parsing was only recently formalized as a task. Different inputs are served by different models with different output formats, although what users want is uniform: what the input contains, and where or when. Unification hits two obstacles. Omni-modal models encode vision and audio in two separately pre-trained towers, so perception has neither cross-modal fusion nor a shared timeline. And ten tasks with heterogeneous output fields resist a single training objective.
The design compresses to one encoder and one schema.
Youtu-Omni-Encoder replaces the vision-plus-audio towers with a single shared Transformer initialized from Qwen3-0.6B-Base. Text-LM initialization preserves weights that already map context to symbols, and parsing is precisely about reading out text, formulas, SMILES, and music notation; only lightweight input stems and output projectors remain modality-specific. Positional encoding does the central work: the rotary frequency budget splits into T/H/W segments, vision tokens take t from the frame index and audio tokens from the time offset, so a frame and the sound of the same instant share one time axis. A few fusion layers then let frames attend to concurrent audio inside temporal attention windows, moving audio-visual alignment from the language model into perception. OmniSchema gives all ten tasks one JSON shape: image tasks emit elements with boxes normalized to 0-1000, audio and video tasks emit segments with one-second timestamps and speaker IDs, and every content field is anchored to its source region or interval.
Pre-training runs four stages totaling 2.24T tokens (encoder-only alignment with a frozen decoder, full-parameter native-resolution training, annealing, then a 128k-token, 512-frame long-context extension). Post-training has three steps: SFT for serialization, schema-routed RLVR with GRPO, then OSAD. In OSAD, a student with LoRA adapters trains on its own rollouts; the teacher is a frozen copy of the same checkpoint whose only privilege is seeing the reference parse, and its next-token distribution on each generated prefix supplies the target. Because outputs follow OmniSchema, tokens carry roles: content tokens get symmetric JS divergence, structure tokens (JSON keys and punctuation) get forward KL at a higher temperature. At each round boundary the merged student becomes the next teacher, so supervision and parser co-evolve; six rounds, no hand-designed reward, no separately trained teacher.
| Benchmark | Metric | Youtu-Parsing-Omni | Strongest baseline |
| OmniDocBench v1.6 | Overall | 96.96 (rank 1) | TeleOCR 96.91 |
| OmniParsingBench | 5-split avg | 75.08 | Gemini-3-Pro 77.44 (API); Logics-Parsing-Omni 72.26 |
| ChemOCR | [email protected] exact match | 54.2 | DeepSeek-OCR 2 49.6 |
| PDMX-Synth (sheet music) | CER, lower better | 22.73 | specialist LEGATO 23.30 |
| PureDocBench | Avg3 | 76.29 (best end-to-end) | TeleOCR 78.41 (multi-stage) |
The margin is uneven. On the three temporal splits it leads Logics-Parsing-Omni (30B MoE decoder) by 2.5 on audio, 6.8 on natural video, and 10.3 on text-rich video; on static splits it trails, 62.84 on natural images (-4.7) and 74.33 on geometry (-2.0). Chart parsing hits 95.02, the highest of any system including Gemini-3-Pro. Two ablations matter most. At a matched 300B-token budget, text-LM initialization buys the shared encoder +130.6 OCRBench points (767.0 vs 636.4), while the from-scratch variant leads BLINK, a spatial-relation benchmark, by 9.5: initialization transfers reading skill, not spatial reasoning. And of the 1.99 points gained after SFT on OmniDocBench, OSAD contributes 1.75 (88%) against 0.24 from RLVR. On Video-MME-v2 (no subtitles), its L1 score of 20.0 beats every non-thinking baseline by at least 5 points; on the L2/L3 reasoning levels it only matches non-thinking peers.
Code and weights are public, and one dense 5B checkpoint covers ten tasks that previously required a shelf of OCR, music-transcription, and chart-recognition specialists. The output is anchored JSON rather than free text, so it drops into RAG and agent pipelines with source traceability. The document SOTA itself deserves a calm reading: the margin over TeleOCR is 0.05 points, closer to parity than a breakthrough. The real gain is that audio and video were added without giving up document depth, and the advantage lands exactly where a shared timeline should help: temporal localization in natural video scores 63.68, 4.21 above Gemini-3-Pro. The OSAD recipe also transfers to any structured-output model where a scalar reward is hard to define.
The paper's own admissions: no inference acceleration yet (the predecessor reused encoder features across regions for parallel decoding); ASR below the compared models throughout (77.94 on the audio split, lowest of the group; 67.80 on natural video, about 16 points behind Gemini-3-Pro); weaker knowledge recognition on natural images (13.90 vs 21.39) and geometric relations (46.77 vs 64.24) against Logics-Parsing-Omni; and no edge on Video-MME-v2's L2/L3 levels without a thinking mode. A further caveat from the evaluation setup: OmniParsingBench shipped without a complete toolkit, so the authors built their own scorer, excluded the camera-video subset plus 136 samples, and corrected some ground truth. The numbers are internally comparable but cannot be checked against the benchmark's published figures, so the 75.08 vs 72.26 lead deserves a discount, as does the 0.05-point OmniDocBench lead within leaderboard noise.