Stanford Paper Introduces CollabSkill: Evaluating Human-Agent Collaboration
paraschopra · x · 2026-07-30
Stanford University and other institutions released the CollabSkill framework, designed to evaluate human-agent collaboration on real-world occupational tasks.
- Background: Existing AI benchmarks mostly focus on fully autonomous model performance, neglecting the human-agent collaboration paradigm, which better preserves human agency and generates economic value.
- Mechanism: The framework pairs real human workers with AI agents and employs a Bayesian skill rating system to disentangle and quantify the respective skill contributions of humans and AI.
- Key Findings: Based on data from 93 workers across 386 sessions, the study reveals that collaboration rankings diverge significantly from fully autonomous benchmarks (e.g., Claude Code ranks first in collaboration, whereas Codex leads in autonomous benchmarks). Furthermore, practical hands-on experience is the primary driver of collaboration skill.
Related event: Human Baselines Missing in AI Evaluations, Highlighting Human-AI Synergy(4 posts)→
More from coding & agent
- Reflection's Beam, a 501B-parameter open MoE model with 1M context, lands on Hugging Face under Apache 2.0 — QGallouedec · 2026-10-16
- 'LLM code is sloppy' rebutted with real ffmpeg code: identical if/else branches in a top library — kingslyj · 2026-10-16
- Office-editing tool calls need file, worktree and unit: lessons from dsh-univer-office — FunDistribution6537 · 2026-10-16
- Architect MD workflow: Gemini one-shot outputs that beat Opus, via 5-lens plan critique — oikk01 · 2026-10-16
- Prove a coding agent actually ran the tests: check artifacts, not approvals — ComprehensiveShake76 · 2026-10-16
- Building an acceptance harness for AI spreadsheet edits: allowlists, recalc checks, idempotency — ukdicha-modakkk · 2026-10-16