Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality
minnesotanlp · hf · 2026-10-27
University of Minnesota's MinnesotaNLP released Kolab on Hugging Face: a dataset built from professional screen recordings for benchmarking human-agent collaboration in long-horizon workflows.
Key finding: frontier multimodal LLM agents systematically overestimate collaboration quality and fail to predict expert adoption of their contributions. The benchmark ships metrics for Agent Contribution, Collaboration Satisfaction, and Workflow Outcome, plus collaboration-dynamic measures like burstiness.
More from coding & agent
- Claude spots broken physics in AI videos, fixes them with CFD and finite-element verification — OfirPress · 2026-10-27
- One prompt: Claude Opus 5.5 adds drone flight mode to RV travel game — ezshine · 2026-10-27
- Do canary tokens still work in the coding-agent era? Fake AWS keys may lure agents — kuza55 · 2026-10-27
- Veteran engineer on what the AI-coding generation will never feel: 36-hour sprints and elusive bugs — bendee983 · 2026-10-27
- Claude Opus 5.5 builds a browser open-world island by comparing screenshots to 341 real photos — prasenx · 2026-10-27
- BreakingWeb benchmark: browser agents fail about half of controlled task variants — shuyanzh36 · 2026-10-27