MOSS-VL Technical Report
Pengyu Wang, Chenkun Tan, Shaojun Zhou, Qirui Zhou, Yanxin Chen, Xingyang He, Huazheng Zeng, Jijun Cheng, Chenghao Wang, Xiaomeng Qian, Pengfei Wang, Zhan Huang, Shanqing Gao, Wei Huang, Longjun Cao, Wu Ran, Jie Liu, Changtai Zhu, Hongkai Wang, Yixian Tian, Chenghao Liu, Zhen Ye, Xinghao Wang, Botian Jiang, Guoguo Feng, Zhaoye Fei, Ruixiao Li, Mingshu Chen, Yang Gao, Qinyuan Cheng, Shimin Li, Xipeng Qiu
cs.CV
2026-08-15
The 11.3B open MOSS-VL family builds perceive-while-generating into the architecture: visual tokens never enter the decoded sequence and arrive as cross-attention cache appends. It posts 66.0 vs 37.5 on OmniMMI Proactive Alerting and widens its time-to-first-token lead over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows.
Open video understanding models are mostly offline: watch the finished clip, then answer. The settings where video understanding matters most do not wait. Live commentary must fire at any moment, a conditional alert must speak in the second the event happens, and the scene keeps moving while the model talks. Existing streaming models (levels L2-L4 in the paper's hierarchy) handle continuous input, deliberate silence, and persistent queries, but share one flaw: they are blind for the duration of each reply. New frames cannot enter, and a wrong statement cannot be revised mid-sentence. Real-time interaction is L5, perceiving while generating, revising or cutting short the moment evidence changes.
The obstacle is architectural. In mainstream designs visual tokens are concatenated into the text sequence and decoded together, so every arriving frame forces recomputation that interrupts the decoding state.
MOSS-VL, built on the Qwen3-8B language backbone with 11.3B total parameters, moves vision entirely off the decoding path:
Training runs a four-stage pre-training curriculum (about 1.26 trillion tokens, sequences growing 8K to 256K, built on large-scale synthesized caption, OCR, grounding, and temporal-grounding data), then SFT produces MOSS-VL-Instruct, and one light Realtime-SFT stage (34.8B tokens, under 3% of the total) installs every real-time behavior. It adds just two state tokens (<|silence|> and <|response|>), puts a decision slot after every frame, and treats speak-or-wait as ordinary next-token prediction, with no dedicated decision head. Of 2.2M emission decisions, 58.7% are self-timed rather than prompted; 5.1% of samples have all-silence as the correct answer. Focal weighting plus inverse-frequency coefficients flatten the class imbalance, and in streams the assistant's turn-final token is excluded from supervision so the model never learns to wrap up just because a user turn appeared; that single change raised emission frequency 39% and mean reply length 68%. One shared system prompt switches among offline, streaming, and real-time modes with a single set of weights and zero architectural change.
Across four streaming benchmarks, MOSS-VL-Realtime posts the best average among open-source streaming models on three (OVO-Bench 70.2 vs 65.3, OmniMMI 32.7 vs 25.4, ProactiveVideoQA 47.2 vs 42.7) and second on StreamingBench visual (69.7 vs AURA's 71.1). Where the gaps sit matters more:
| Proactive subset | MOSS-VL-Realtime | Best baseline |
| OmniMMI Proactive Alerting | 66.0 | 37.5 (AURA) |
| StreamingBench Proactive Output | 60.0 | 53.2 (AURA) |
| OVO-Bench Forward Active Responding | 62.1 | 55.8 (AURA) |
| OVO-Bench Backward Tracing | 72.6 | 60.4 (AURA) |
All three subsets that squarely test speaking unprompted at the right moment go to MOSS-VL; where a subset reduces to perception QA over the current scene, AURA keeps the edge.
Efficiency is measured on SGLang, one H200, BF16, not estimated. Against Qwen3-VL-8B, which shares the same Qwen3-8B backbone, the time-to-first-token gap widens from 2.8x to 5.1x as visual context grows, and end-to-end latency from 1.9x to 4.3x with ViT output matched. The same-video comparison is stricter: MOSS-VL forgoes temporal compression and carries roughly twice the vision tokens, yet never falls behind.
Offline, MOSS-VL-Instruct trades wins and losses across 39 benchmarks. It takes five of twelve perception rows (BLINK 78.0 by 8.9 points) and leads every temporal-reasoning video set, Minerva 40.5, TOMATO 39.5, VideoMME-Logical 17.1; it trails on MMMU, document suites, and standard grounding, which the authors attribute to shipping without a thinking mode and optimizing for real-time video rather than exam-style reasoning.
For anyone building video-interaction products, this is the first open, full-stack reference for L5 behavior: weights, the staged curriculum, real-time inference code, and an extended FlashAttention-3 backend all released. The architectural lesson transfers to any system that must absorb a new modality stream mid-generation, whether speech, sensors, or logs: keep those tokens out of the decoded sequence behind an append-only cross-attention cache and the latency-versus-context curve flattens immediately. Realtime-SFT is also a bargain: two vocabulary entries and a loss reweighting install the interaction behavior with no RL and no new head.
The 66.0-versus-37.5 margin on Proactive Alerting shows that knowing when to speak is directly supervisable and teachable, not a byproduct of scale.
The authors' own list: MOSS-VL-Instruct trails the strongest open peers on MMMU and document benchmarks and ships without a thinking mode; the headline L5 behavior, revising a reply mid-generation, is attested only qualitatively through live demos and the released code, because quantitative validation stops at L2-L4 where public benchmarks end, and the field lacks a benchmark for perception during generation; RL post-training is on the roadmap but not done.
Further concerns from a close read. Most baseline streaming numbers come from the respective official reports with inference settings, notably frame counts, that may not match MOSS-VL's, which the paper admits; read cross-model scores with care. The 11.3B-versus-7-8B comparison is defended on the grounds that the extra parameters sit outside the decoded sequence and the language backbone is same-class, but the larger vision encoder and cross-attention stack remain an uncontrolled factor. The synthesized interaction corpus comes from a caption-driven pipeline with frame-verified temporal anchors, yet the realism of revision behavior relative to genuine multimodal dialogue has no external validation.