TRACE benchmark shows near-identical QA scores mask big gaps in streaming video models
omlab · hf · 2026-09-28
omlab released TRACE (Temporal Audit and Condition-aware Evaluation), a condition-aware benchmark for streaming video understanding that makes evidence timing, history maintenance, and response triggering explicit.
Why: current evaluations report task scores without specifying when evidence becomes valid or how responses are triggered, so similar scores can hide very different workloads and failure modes.
How: TRACE combines temporally audited tasks with evidence-timing and trigger annotations, a causal Core–Adapter protocol that controls information availability while logging actual history processing and response events, and multidimensional reporting (answer quality, timeliness, response-selection behavior, workload, completion, reliability).
Findings: across 1,240 records from 517 videos and 8 models in 8 configurations, nearly identical QA accuracy masked large differences in completion, answer validity, and generation workload. Proactive performance further splits into response quality, delay, false alarms, and missed target windows. Code and benchmark are open-sourced on GitHub.
More from Multimodal
- Stunning AI-generated motion-blur portraits with a film-photography feel — azed_ai · 2026-11-16
- A Bauhaus functionalist prompt template for geometric poster-style AI art — LudovicCreator · 2026-11-16
- Vivix W1 video model: 100 ten-second ads for about $5 at $0.005 per second — Aiden_Tech_Ai · 2026-11-16
- User Shares Coral Reef Creation Labeled 'Qwen 4 Max (xhigh)', an Unannounced Model Name — intellectronica · 2026-11-16
- Reddit megathread crowdsources the best open-source audio models: ASR, TTS, and text-to-music — rm-rf-rm · 2026-11-16
- Animating AI-generated 2D game art with FLUX.2 klein: masked redraws work but remain a hack — emilrueh · 2026-11-16