TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming
Yibo Hu, Yu Qian, Mao Gu, Yingfan Tao, Yuhao Chen, Yongdong Luo, Zhuoqun Liu, Meiguang Jin, Junfeng Ma
cs.AI, cs.CV
2026-08-21
TLive-Omni wires Qwen3-Omni audio into Qwen3.5, aligns video grids with audio via Per-vGrid, then scores final answers with Faithful-RFT. 4B product AP 91.45; 9B live ASR CER 6.46.
Live commerce scatters product facts across host speech, camera frames, product cards, overlay text, and viewer questions. Those channels often fail to line up. The host says "this one" after the camera has already cut away.
General omni models can ingest image, video, and audio, but their data and evals are not built around products. MiniCPM-o 4.5, Qwen3-Omni, OmniVinci, and Nemotron 3 Nano Omni target open-domain interaction. ByteDance's Valley3 moves toward e-commerce, still without a fine-grained live-streaming skill list. TLive-Omni, from Alibaba's Taobao & Tmall Group, is narrower: understand a long, noisy live stream, and keep the answer tied to the evidence.
The backbone is Qwen3.5. Output is text only, with a 256K multimodal context. Vision uses the native Qwen3.5 pipeline; after spatial merge an image becomes (h/32)×(w/32) tokens. Audio is not an external ASR dump. The model takes Qwen3-Omni's AuT encoder, trained on about 20 million hours of audio, 16 kHz 128-dim mel, about 13 tokens per second, then a two-layer aligner into the language model. Offloading transcription would drop time alignment and speaker cues, so audio stays a first-class input.
Per-vGrid is the token layout. Each video time grid sits next to the audio from the same interval, wrapped by boundary tokens, with a textual timestamp computed from the frames that were actually sampled. Relative to Qwen3-Omni's audio-video packing, this adds explicit timestamps, grid boundaries, and a hard contiguity constraint inside each grid. Requesting 2 FPS is not the same as getting 2 FPS. A 119-frame clip at 30 FPS sampled at a requested 2 FPS can land near 1.76 FPS; a grid then spans about 1.13 seconds instead of 1.0, and the audio span grows from about 13 tokens to 14-15. Timestamps follow the realized sampling, so audio and video stay aligned.
Supervised training has three stages. Stage 1 freezes the language model and audio encoder, and trains only the aligner on 5M ASR samples. Stage 2 trains the audio encoder and aligner on 26M ASR, caption, and audio-QA samples; the language model stays frozen. Stage 3 jointly fine-tunes 14M multimodal samples. Encoders stay frozen; aligners and the language model train on transcription, speaker analysis, product grounding, OCR, temporal grounding, dense caption, and omni QA.
Faithful-RFT then runs GRPO with 8 candidates per prompt. Rewards score the final answer with task-checkable signals and suppress explicit think tags. Live understanding wants a faithful answer quickly; it does not pay for reasoning length. Rewards route by task: rules for multiple choice, boxes, and OCR; an LLM judge for open-ended answers; a format penalty only to ban think tags. Groups with near-zero reward variance are regenerated so GRPO still has a relative signal. A synchronized length-grouped sampler cuts padding and keeps worker load even.
The data engine builds audio, image, and video tracks separately. Audio uses an ASR ensemble plus a domain keyword list. Speaker labels are cross-checked between acoustic diarization and a multimodal predictor, with lip motion as a tie-breaker. Product boxes come from a detector-judger loop. Dense captions split on physical shots; QA and temporal grounding split on semantic events.
Live-commerce numbers come from an in-house Taobao suite. On ASR, TLive-Omni-9B posts a CER of 6.46 and the 4B model 6.66, both below open-source Qwen3-Omni 30B-A3B at 6.75 and closed Gemini 3.5 Flash at 13.09. Speaker-attributed cpWER is 12.27 for the 9B model against 27.84 for Qwen3-Omni. Audio-description accuracy sits near 76, still behind Gemini 3 Pro at 85.07.
The image gap is larger. Product-image [email protected] is 91.45 for 4B and 89.96 for 9B, against 68.88 for Qwen3-Omni and 74.89 for Gemini 3.5 Flash. Live-frame AP is 82.85 for 4B, close to Gemini 3.5 Flash at 84.15. OCR localization F1 is about 87, recognition NED is 4.72 / 4.24, and classification accuracy is about 80. Those three beat every open and closed model in the comparison.
Video temporal-grounding mIoU is 81.49 for 9B, versus 77.90 for Gemini 3 Pro and 39.22 for Qwen3-Omni. Dense-caption accuracy is 74.63 with an 8.76 hallucination rate, against Gemini 2.5 Flash at 54.60 / 10.97. Video QA is 93.23 for 9B, a shade above Gemini 2.5 Pro at 92.62.
General benchmarks are not gutted by the vertical training. The 9B model leads open-source numbers on MMBench (88.9), MLVU (80.9), Video-MME (75.6), and LongVideoBench (69.9). The 4B model actually tops the 9B on TimeLens mIoU (57.0 / 58.2 / 69.2) and VideoMMMU (73.9). On the omni set AVUT, 9B scores 80.0, above MiniCPM-o 4.5 at 78.6 and Qwen3-Omni at 74.2.
| Task | TLive-Omni-4B | TLive-Omni-9B | Reference |
| Live ASR CER ↓ | 6.66 | 6.46 | Qwen3-Omni 6.75 |
| Product [email protected] ↑ | 91.45 | 89.96 | Gemini 3.5 Flash 74.89 |
| Live TG mIoU ↑ | 77.63 | 81.49 | Gemini 3 Pro 77.90 |
| Video-MME ↑ | 71.3 | 75.6 | MiniCPM-o 4.5 70.4 |
This is a vertical omni stack whose architecture, data, objective, and eval all sit on live commerce. It is not another general omni release. Product-image grounding, price-tag OCR, and SKU-name ASR are still far from usable in open omni models. TLive-Omni moves them into a range a business team can actually run. 4B and 9B weights are on Hugging Face.
Faithful-RFT is a clear trade: score the final answer, keep latency down, do not reward chain-of-thought. The length-grouped sampler and zero-variance resample are GRPO plumbing that other vertical teams can copy.
The model understands. It does not speak, and it is not full-duplex. A live assistant still needs a separate generation and interaction stack.
The authors say so: understanding only, no generation or full-duplex interaction; public-benchmark coverage should grow; robustness on longer, noisier, more diverse streams is unfinished; temporal calibration under missing evidence is weak.
Every live-commerce number comes from an in-house suite. Outsiders cannot rerun it. The 4B model beats the 9B on product AP, HallusionBench, TimeLens, and VideoMMMU, which may be eval noise or a smaller model hugging the vertical data harder. On MMMU the 9B scores 73.4, a hair below the Qwen3.5-9B backbone at 74.2. SimpleVQA is 50.0 against Gemini 3 Pro at 73.2. Killing think traces is a plausible cost on hard reasoning. Audio description still trails Gemini 3 Pro by a wide margin. On shot layout, Gemini 2.5 Pro remains ahead.
The data engine and the internal eval set are not released as reproducible assets. What is confirmed: on Taobao's own ruler, the vertical model beats general omni systems. Whether the ruler is fair is still unknown.