Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition
Yuhao Dong, Shulin Tian, Shuai Liu, Shuangrui Ding, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang, Ziwei Liu
cs.CV
2026-02-09
HowTo100M becomes 1,200 next-step questions after a text or video demo. Open 7B models barely gain from video demos; Demo-ICL lifts text-demo accuracy from 31.4% to 43.4%.
Most video benchmarks ask for facts the model already knows, or facts sitting on screen. "What is a whisk" and "where is the whisk" can both be right, and still say nothing about whether the model can watch a Mexican-rice tutorial and then, on a new kitchen clip, decide the next step.
That is closer to how people learn a procedure, and closer to robot learning from demonstrations. NTU S-Lab and Shanghai AI Lab isolate that skill as Demo-driven Video In-Context Learning: acquire a procedure from a text or video demonstration in context, then answer questions on a target video.
The eval has three rungs. Text-demo ICL supplies a written recipe. Video-demo ICL supplies a related tutorial video. Demonstration selection first picks the right tutorial from a distractor pool, then answers, which is closer to a messy retrieval setting.
Demo-ICL-Bench is built on HowTo100M, with WhisperX timestamps from HTM-AA. Text demos go coarse-to-fine: Qwen2.5-72B turns ASR into steps, Qwen2.5-VL-72B rewrites them against the matching clips. Video pairs start from YouTube search rank and title similarity, then an LLM checks that the generated instructions describe transferable tasks. Questions sit on an intermediate step (never first or last); the text track requires at least six steps and asks what comes next. After human filtering the three tracks keep 500/500/200 items, 1,200 in total. A text-demo spot check passes 96% of items; humans can confidently pick the right demo in 88% of selection cases.
Demo-ICL is an Ola-Video descendant (OryxViT plus Qwen2.5) trained in two stages. Video SFT mixes LLaVA-OneVision, LLaVA-Video, Oryx, Ola, plus COIN and Cross-Task, and a subsample of demo-ICL items with benchmark videos held out. Stage two is information-assisted DPO: chosen answers are generated with privileged hints (timestamps for text demos, paired text steps for video demos), ordinary answers are rejected, then the loop iterates. Training uses 64 A100s, batch 256, SFT learning rate 1e-5, DPO 5e-7, and 5,000 DPO pairs. Evaluation samples 32 frames and withholds subtitles.
Humans average 80.1% on Demo-ICL-Bench. Gemini-2.5-Pro falls to 38.9% (54.4 text-demo, 36.2 video-demo, 26.0 selection). GPT-4o sits at 34.9%. Open 7B models are weaker, and video demos often hurt.
| Model | Text-demo | Video-demo ΔICL | Avg |
| Human | 84.0 | n/a | 80.1 |
| Gemini-2.5-Pro | 54.4 | not reported | 38.9 |
| Qwen2.5-VL-72B | 45.0 (+20.8) | +0.4 | 29.5 |
| Ola-Video 7B base | 31.4 (+8.6) | -1.0 | 24.8 |
| Demo-ICL 7B | 43.4 (+14.0) | +4.4 | 33.1 |
Scale helps text demos a lot: the 72B model is no better than 7B without a demo, then jumps 20.8 points once the recipe is in context. Video demos barely move. Demo-ICL's 7B average of 33.1 beats Qwen2.5-VL-72B's 29.5, mostly on video-demo (32.0 vs 25.6) and selection (24.0 vs 18.0). Its text-demo 43.4 still trails the 72B's 45.0 and Gemini.
General video scores hold. VideoMMMU goes from 46.2 on the base to 52.6, above Qwen2.5-VL-7B at 47.4. Video-MMLU overall is 45.7, Quiz 50.4, against 37.5 / 32.9 for Qwen2.5-VL-7B. VideoMME without subtitles is 65.2, in the same band as other 7B models.
Ablations pin the hard part. Raising video-demo from 32 to 128 frames moves 29.4 to 30.4. Replacing the reference with the target video itself (full-video peek) jumps to 38.6; next-step clips reach 35.8; ASR plus captions reach 45.4. Perception is easier than abstracting a procedure from another video and transferring it. On the training side: SFT without instructional videos averages 26.4, SFT 29.8, vanilla DPO 30.7, one round of privileged DPO 31.7, the full loop 33.1.
The paper pulls "can a video MLLM learn on the spot" out of ordinary zero-shot QA and gives it a reproducible exam. The useful takeaway for robot imitation and tutorial assistants is blunt: written recipes are barely usable, video tutorials are not teaching material yet. A 7B model beating a 72B average after a data recipe and DPO also says this skill is sensitive to the training signal, not just parameter count.
It is not ready to use. 33.1 versus Gemini 38.9 versus humans 80.1, with video-demo still the bottleneck. If a product assumes the model can watch a tutorial and then execute in a new scene, this paper is a counterexample, not a recipe.
The authors note two gaps. There is no architecture built for demo ICL, only a training recipe. They also never test mixed context, text plus video at once, which is how people actually follow a tutorial.
A few issues get less airtime. Chosen DPO answers are written with hints the model will not have at test time, so the update may teach "how to write when clues are present" more than "how to abstract a demo". On demonstration selection, Qwen2.5-VL scores only 22.2% even after picking the right video, below 25.4% when that video is given directly; extra candidates scatter attention. Questions are LLM-generated; about 4% of text-demo items fail a human check. The domain is HowTo100M household procedures. Lectures, lab work, and factory assembly may not fail the same way. The abstract quotes Gemini at 46.6% / 32.0% on text / video demos; Table 1 has 54.4 / 36.2. Use the table.