CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks
Yijia Shao, Zora Zhiruo Wang, Neel Ahuja, Yicheng Wang, Bowen Liu, Diyi Yang
COLM 2026
cs.HC, cs.AI, cs.CY
2026-04-20
Pairing 93 real workers with AI agents across 386 sessions, this COLM paper finds the best solo agent is the worst collaborator, Claude Code beats Codex, and experience, not attitudes, drives human skill.
Almost every agent benchmark lets the agent work alone. SWE-bench has it fix bugs by itself; GDPval has it build spreadsheets by itself, and whoever scores highest wins. Real work is rarely like that. An agent is paired with a person who gives instructions, fills gaps, and course-corrects. Being strong alone says little about being strong as a teammate.
The harder problem is that people vary enormously. Hand the same prompt to a veteran and a novice and the outcome can look completely different. Averaging scores to rank agents gets contaminated by that variance: an agent may score low because it drew weak partners, not because it is weak. Prior collaboration studies either simulate users with an LLM (too cooperative, stylistically uniform, devoid of real frustration) or use tasks too simple to resemble real occupations.
CollabSkill does three things.
First, it gathers real data. The authors pick tasks from GDPVal, APEX, and APEX-Agents spanning 10 of the 20 ONET occupational sectors, recruit 93 US workers through Upwork (averaging 9.6 years of experience), match each to a random agent, and have them work locally, submitting deliverables and interaction logs. This yields 386 sessions and over 1,500 prompts, with a median session length of 76.7 minutes.
Second, it grades the output with a two-stage automated grader: generate a rubric, then have multiple LLM judges score against it.
Third, the core contribution: a Bayesian rating system that separates human and agent contributions. Each teamwork outcome is modeled as agent skill plus human skill plus noise: y = sA + sH + ε. Every entity gets a Gaussian prior, and each new observation updates the posterior via a Kalman filter. The final score is μ - 3σ, the mean minus three standard deviations, which deliberately penalizes entities with high uncertainty. This is the conservative-rating idea behind TrueSkill. Rankings then reflect each agent's collaboration ability rather than being dragged up or down by whoever it was paired with.
The ranking almost inverts the solo leaderboard.
| Agent | CollabSkill | Solo | SWE-bench Verified |
| Claude Cowork | 76.738 | n/a | n/a |
| Claude Code | 74.778 | 77.17 | 71.40 |
| Codex | 71.267 | 78.02 | 72.80 |
| Manus | 71.204 | n/a | n/a |
| Gemini CLI | 69.555 | 79.44 | 69.60 |
On the solo leaderboard Gemini CLI is highest (79.44), Codex next, Claude Code lowest. On the collaboration leaderboard Gemini CLI is dead last. Restricting to the three terminal-based agents for an apples-to-apples comparison, Claude Code ranks first, Codex drops to third, and the probability that Claude Code beats Codex exceeds 0.999. This reverses SWE-bench, GDPval, and SciCode, where Codex consistently leads Claude Code.
Interface design has its own effect. Claude Cowork and Claude Code run the same Claude model; the only difference is that Cowork is a desktop graphical interface for non-developers while Code is a terminal. On that alone, Cowork ranks strictly above Code (p > 0.999).
On the human side the verdict is blunt: hands-on experience drives collaboration skill, not attitudes. Self-reported LLM familiarity correlates with skill (Spearman ρ = 0.297, p = 0.010), as does comfort delegating to an agent (ρ = 0.238, p = 0.041); no other attitudinal measure reaches significance. The skill spread is wide: splitting workers into quartiles, the top quartile (Q4) wins 74% of the time against the autonomous baseline, the bottom quartile (Q1) only 27%. After one collaboration session, trust in agents rises (Δ = 0.58, p = 0.004) and so does delegation comfort (Δ = 0.45, p = 0.039). Workers also revise their belief about how much autonomy agents deserve upward (Δ = −0.50, p < 0.001), though the autonomy level they personally want does not change (p = 0.105, not significant).
The practical takeaway: do not pick agents off solo leaderboards alone. The agent that tops a solo benchmark can fall to last place inside a real workflow. If the team is mostly non-developers, a GUI agent like Cowork is worth more than a bare terminal. And for people, time on the tool matters more than changing attitudes, which points somewhere concrete for training and rollout.
More fundamentally, the paper offers a methodological template: model human and agent skill explicitly and estimate them separately, and you can produce a trustworthy agent ranking even when people vary wildly. It was never that collaboration could not be measured; it was that no one had removed the human noise.
The sample is biased. All 93 workers are US-based Upwork freelancers covering only 10 sectors, which rules out fine-grained subgroup analysis. Grading relies on LLM judges rather than human graders (human grading was prohibitively expensive), so judge bias is not fully ruled out. The model splits outcomes into agent skill plus human skill but does not further separate task skill from collaboration skill, so an agent that is strong at the task but bad at cooperating gets conflated into one number. The 9.6-year average experience is also high, likely overstating how typical workers collaborate.