OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs
Xianyun Sun, Chaoyou Fu, Zhengye Zhang, Feiyang Duan, Qingyuan Cao, Yonghui Niu, Sihang Yuan, Ge Zhang, Caifeng Shan
cs.CV
2026-08-22
OmniAssistBench reverse-engineers 300 internet videos into 685 assistant-style turns. Gemini-3-Pro scores 66.4/100; the best open model, Qwen3-Omni-Instruct, reaches 51.2.
Video-understanding benches treat a clip as a frozen exam. The model's answer does not rewrite the next frame. A live assistant works differently. If it says "foam the milk first," the user foams the milk, and the camera follows. The same goal can fork into several valid paths. An offline dataset cannot stock a follow-up video for every fork.
That is why static multiple-choice tests miss assistant skill: the reply itself scatters the trajectory. OmniAssistBench, from Nanjing University, Nankai University, and the University of Waterloo, locks the path with priors mined from the source video, then reverse-engineers internet footage into multi-turn assistant chats.
Annotators extract one canonical procedure from each source clip, such as "pull espresso before steaming milk," and hand it to the model as a prior. The model must shepherd the user along that route. Alternative procedures that would also make a latte are scored as wrong. With one ground-truth path, offline grading becomes possible.
Construction is reverse engineering in four stages:
The set has 300 videos and 685 QA pairs, 182 seconds on average, built with more than 1,000 expert hours. The basic tier covers social perception, non-salient temporal detail, referential disambiguation, and non-audio prompts. The advanced tier covers grounded advice, delayed response, and multi-step process tracking. Three filmed real-world cases add a 20-minute 12-person meeting, blind navigation in an office, and a three-person handicraft session, about 15 turns and 16 minutes each.
Non-temporal clips are trimmed to 30-180 seconds at about 1080p. Gemini models see native resolution; others are sampled at 1 fps. Overflowing context drops the oldest turns FIFO. The system prompt requires the exact token [KEEP QUIET] when the model should stay silent.
Gemini-3-Pro leads at 66.4. The strongest open model, Qwen3-Omni-Instruct (30B-A3B), scores 51.2. Under the rubric, current models usually parse spoken prompts and still fail to give complete, correct help.
| Model | Overall | Basic tier | Real-world |
| Gemini-3-Pro | 66.4 | 63.6 | 68.0 |
| Gemini-2.5-Pro | 64.6 | 65.4 | 44.8 |
| Doubao-Seed-2.0-lite | 57.3 | 53.2 | 35.5 |
| Qwen3-Omni-Instruct | 51.2 | 46.4 | 53.8 |
| MiniCPM-o-4.5 | 46.0 | 44.6 | 37.8 |
| VITA-1.5 | 24.6 | 24.6 | 14.6 |
Gestures are a shared failure. On Gesture-based Prompt Following, Gemini-3-Pro scores 55.0, Doubao 29.5, Qwen3-Omni 28.2. Multi-event delayed response (MER) is 48.4 even for Gemini-3-Pro. On the filmed cases Gemini-3-Pro beats Gemini-2.5-Pro by 23.2 points, and still loses small objects once they leave the frame. Most models cannot bind a hand-drawn map to the office in the blind-assistance case.
Ablations: dropping audio or video hurts most tasks, yet delayed-response scores rise in the vision-only setting because background speech no longer masquerades as a new user turn. Teacher forcing, swapping history for gold answers, barely moves the needle: models copy the terse format and still fail to ground it in the video. Shrinking frames from 1080p to 360p buys about 50 extra seconds of context and almost no benchmark gain. GPT-5, GLM-5, and DeepSeek-v3.2 judges agree, with Pearson correlations above 0.75.
If you ship a real-time video assistant, this table sits closer to the product than Video-MME-style frozen quizzes. Gesture following, knowing when to stay quiet, and keeping the original goal across turns are all unsolved. Open models are not usable assistants yet. Closed models have only just crossed the "partially valid" line.
The dataset is expensive: 1,000 hours for 300 clips. The transferable piece is the evaluation trick, locking a path with a prior and simulating online chat by editing existing video.
A locked path tests "follow this recipe," not "pick a good route among several." Real assistants often have to plan. User goals are inferred after the fact from internet video, and may not match how people actually ask for help. Three real-world cases is a thin sample; rank gaps will be noisy. The judge is GPT-5; high correlation across judges does not rule out a shared bias. Gemini and the other models see different frame rates, so protocol leak may sit inside the scores. No human ceiling is reported.