Stanford benchmark: frontier models reach best team deal in only 14% of conflicts
rohanpaul_ai · x · 2026-11-05
Stanford (with MIT and Oxford) released the Collaborative Intelligence Benchmark, measuring how AI agents perform as teammates — filling a gap left by benchmarks that score solo or whole-team results. Key points:
- 240 tasks across 8 teamwork dimensions (aviation reports, clinical cases, coding, negotiations), with scripted teammates creating conflicts; models are scored only on tasks they can solve alone, isolating collaboration from raw capability.
- Best of 18 models scored just 82.2/100; frontier models ask about teammates' needs in 99% of conflict runs yet accept the weaker offer 81% of the time, reaching the best team deal in only 14% of conflicts.
- Models almost always refuse unsafe shortcuts and report misconduct, but cave under interpersonal pressure: pass rate drops from 97% to 81% when taking over tasks where a weaker model cheated.
- Models with nearly identical capability scores differ by up to 11 points as teammates — general capability doesn't guarantee teamwork.
The authors suggest testing models against pushy teammates and inherited work before adding them to agent teams. Includes 160 public tasks (code and data) plus 80 held-out tasks. arXiv: 2611.02633.
More from AGI Musings
- Sahil Lavingia on bringing AI-assisted development to the IRS app — every · 2026-11-05
- Miessler: a drug boosting empathy and self-restraint 10-25% may be the AGI Holy Grail — JensHonack · 2026-11-05
- 320M Uber trips in Halloween week, only 4 booked by an agent: 'Agentic commerce is a mirage' — ivan_bezdomny · 2026-11-05
- DeepMind philosopher says autonomous AI agents keep emailing him — about philosophy and job hunting — ShaneLegg · 2026-11-05
- AI agent audits mom's APOE E3/E4 genetic report, with a second AI trying to break the work — kevrussell · 2026-11-05
- Kevin Russell launches Kram, a public AI map of Alzheimer's research, as his mom enters hospice — kevrussell · 2026-11-05