StreamEval: Evaluating Interactive Streaming Video Generation Models
StreamEval Team
cs.CV
2026-10-20
StreamEval scores 37 models on 300 sessions. Quality and identity can be high; avatar action following peaks at 36.9 and full reference-edit success at 15.
Streaming generators take new input while frames are still coming out. An avatar has to continue a conversation, a world has to absorb a new object, action, or camera, and a live editor has to change one region of a source that keeps advancing, then swap the reference and later undo the change. Each request carries a time, a target, and a list of things that must survive: identity, scene, and edits already made.
Current scores split that contract. VBench and EvalCrafter measure look and temporal behavior. T2V-CompBench measures compositional alignment. WBench covers multi-turn control inside generated worlds. StreamAV-Bench rolls streaming audio-video out to about 180 seconds. LongV2VBench stops its editing cases around 60 seconds. VideoFDB scores nonverbal cues in full-duplex conversation. Prescribed body performance, edits on an external source, and avatar dialogue have not shared one protocol that releases inputs on a clock.
StreamEval has five tasks, 60 scripted sessions each, 300 in total: digital-human dialogue, avatar performance, world generation, instruction editing, and reference editing. Durations are 15, 30, 60, 120, and 300 seconds, 12 sessions per bin. The bins have equal weight, so a 300-second case does not outweigh a 15-second one. The pool has 240 source families, plus 60 text dialogues that reuse an identity, and 1,347 scheduled online events. One pass is about 525 minutes of active generation, excluding setup and trailing replies.
A case binds an opening portrait, scene, or source video to a timeline. Speech, text, and reference images arrive only when the clock reaches them. Checkpoints say what to establish, keep, cancel, or restore, so execution and preservation are scored apart. GPT-Image-2 and Qwen-Image supply portraits of synthetic adults or stylized fictional characters. Speech comes from Qwen3.5-Omni and must stay within a normalized word error rate of 0.25. Sources are not looped, sped up, or spliced. Qwen3.8-Max drafts the scripts. A human-authored oracle locks the checkpoints.
Ten metrics use a 0 to 100 scale. Q-Overall is their equal-weight mean, and the active set depends on the task. Dialogue is the share of planned turns that meet answer, content, conversational rules, and timing together. Following is the share of controls that land: gaze, expression, pose, and gesture for performance; in a world, every frame where the target is still visible must show the requested change at least in part; an instruction edit may match appearance only partly when nothing leaks severely. Edit Success is stricter. Scope, deadline, active state, persistence, and protected content all have to hold. Preservation checks regions nobody named, plus the source motion at that moment. An undo returns to the source as it looks then. Continuity checks camera, object permanence, and spatial layout. Quality maps each five-second clip from 100 down to 0 by defect severity, severe defects first. Stability counts only time that also clears the Quality bar, so clean footage is entered twice. Identity keeps the lower mark from Qwen3.8-Max-0902 and Qwen3VL-Plus. Fixed avatars are compared with their own opening frame. Reference is scored only while its image is active. A-V Sync requires lip opening and the audio envelope to correlate at least 0.3, with the delay's 95% interval inside plus or minus 200 milliseconds.
Three runs are averaged, then cases inside each duration bin, then the five bins. Missing output and unsupported operations score 0. Controls that never fire stay in the denominator. Unresolved required judgments enter the tables as 0. Dialogue and performance add 5 when the API accepts a custom avatar, so Overall can reach 105. The bonus does not change the metric values. World generation averages four metrics and gets no bonus. Judge probes were accepted at up to 5% false positives, 10% false negatives, and 10% unresolved calls per class. The paper reports 37 models and 47 task configurations, with three runs per case.
On 15 dialogue configurations, Vidu S2 Avatar scores Overall 93.98, from Q-Overall 88.98 plus 5. Quality is 96.78, Dialogue 75.29, Identity 90.12. AVTR-1 scores 84.14. Tavus CVI Full cannot take a custom portrait and gets no bonus: Q-Overall 86.65, the best A-V Sync at 94.44, and Dialogue 73.91. LiveAvatar FULL scores Stability 100 and A-V Sync 0. MuseTalk 1.5 scores 95.34, 97.11, and 88.44 on Quality, Stability, and Identity, then Dialogue 54.34 and A-V Sync 38.09, for Overall 79.66.
All seven performance models receive the bonus, so it does not reorder them. Vidu S2 Avatar leads with Overall 81.71 and Following 36.90, plus A-V Sync 99.60 and Identity 86.12. Avatar-Forever 22B is last on Following at 6.37. LiveTalk 1.3B has the best Quality at 71.27, Following 16.59, and Overall 75.85. Identity above 80 and Stability above 96 still leave most scripted actions missed.
World-generation Overall peaks at 33.36 for MemFlow, then 33.01 for Rolling Forcing and 31.61 for Krea Realtime. The best Following is Krea Realtime at 25.93, which the abstract assigns to text-conditioned world generation. MemFlow's Following is 22.59. HappyOyster 1.0 Adventure leads Quality and Stability at 58.34 and 56.67, with Following 5.56 and Continuity 2.78. Matrix-Game 3.5 leads Continuity at 34.50.
On instruction editing, Vidu S2 Editing scores Overall 45.43, Following 72.10, Edit Success 57.20, Preservation 60.93, Quality 22.99, and Stability 13.94. JoyAI-Video-Edit reaches Preservation 83.33, Edit Success 43.52, and Overall 42.39. Reference editing shows three models kept because Overall is at least 5. Vidu S2 Editing scores Edit Success 15.00, Overall 39.02, Reference 31.20, and Preservation 64.44. JoyAI-Video-Edit scores 8.89 on Edit Success. XMax X2.0 scores 1.11. In the recorded cape session the reference switches from red velvet to gold satin at 10 seconds. Sampled outputs keep the original grey cape.
| Task | Best | Comparison |
| Dialogue | Vidu S2 Avatar 75.29 | Tavus 73.91, MuseTalk 54.34 |
| Performance Following | Vidu S2 Avatar 36.90 | Avatar-Forever 6.37 |
| World Following | Krea Realtime 25.93 | MemFlow 22.59, HappyOyster 5.56 |
| Instruction Edit Success | Vidu S2 Editing 57.20 | JoyAI-Video-Edit 43.52 |
| Reference Edit Success | Vidu S2 Editing 15.00 | JoyAI 8.89, XMax 1.11 |
The scores separate a finished-looking frame from a session that did what it was told. Action following peaks at 36.90, world following at 25.93, and full reference-edit success at 15.00. Ranking realtime systems by picture quality would move HappyOyster up, with Quality 58.34 and Following 5.56.
The horizon moved as well. Editing runs to 300 seconds, five times LongV2VBench's 60-second tests, and world cases run past StreamAV-Bench's 180-second rollouts. WBench, WorldMark, and StreamAV-Bench had reported 20, 10, and 13 systems in their own settings. Here 37 models share one execution and preservation rubric. The paper does not introduce a new generator. It measures the execution gap.
Ranks come from vision-language judges. Human pairwise judgments, and agreement with this rubric, are deferred. The 5% false-positive and 10% false-negative lines are probe targets. Realized rates on these 300 sessions are not published.
Stability is gated by Quality and then averaged with it, so a clean picture collects the visual term twice. The 5-point bonus does not reorder the dialogue top three: Vidu without it is still 88.98, ahead of Tavus at 86.65 and AVTR-1 at 84.14. The middle does move. Vivix A1's Q-Overall is 61.47, but Overall becomes 66.47 after the bonus, above LiveAvatar FULL at 65.07. Every performance model receives the same 5 points, so that order stays put.
Editing tables keep only configurations with Overall of at least 5, and the cut was chosen after the runs. The published 37 models and 47 configurations refer to that reported subset. An Edit Success of 15.00 is the best of the three reference-editing models on display. A-V Sync samples every 500 milliseconds in dialogue and every 250 milliseconds in performance, which is coarse beside lip error. World models run under both text and image conditions appear only with the text numbers. Faces are synthetic. Broader identities and places, global resets, and undoing one edit while others stay active are marked for later. Perception of a live user video sits outside the protocol.