AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs
Raphael Shu, Yusen Zhang, Young Min Cho, Jin Mo Yang, Yuan Yuan, Wenliang Zheng, Sharath Chandra Guntuku, Lyle Ungar, Zhou Yu, Rui Zhang
COLM 2026
cs.MA
2026-09-26
AgentWorld tests 3–20 blackbox agents over 25–55 MMORPG rounds. Gemini 3 Flash leads at 52.0% success and 0.320 CCE; 27% of tasks are unsolved by all four models.
Most multi-agent LLM benchmarks are short (under 20 steps), competitive, or just sums of single-agent scores. Minecraft-style worlds then spend those scores on locomotion and block placement, so collaboration barely moves the number. Social simulators are lively and have almost no pass/fail tasks.
AgentWorld isolates blackbox, long-horizon collaboration with asymmetric roles. Agents cannot see each other’s internals; they coordinate only through in-world chat. Teams of 3–20 have different skills and spawn points, so no single agent can finish the job. The paper is at COLM 2026, from OpenAgents, Columbia, Penn, Seoul National University, and Penn State.
The sandbox sits on Kaetram, an open-source MMORPG: a 1056×768 map, 9 biomes, 380+ items, 144 mobs, 70+ NPCs, and 13 high-level tools (move, attack, harvest, craft, transfer, chat, …). attackentity wraps pathfinding, combat, and loot, so the score is not a micro-control contest. The live game is discretized into rounds: one tool call per agent per round, fixed order, deterministic replay.
There are 100 human-authored tasks and 100 Claude Opus variants, across combat, crafting, gathering, trading, exploration, survival, construction, and coordination. Most teams are size 3, with a tail to 20. Horizons run 25–55 rounds, mean 38. A Python verifier reads the final game state; partial success rate (PSR) is the fraction of checkpoints hit.
Causal Collaboration Effectiveness (CCE) walks backward from the success actions, asking an LLM only binary questions of the form “did this action enable that later action?”. CCE is contributing actions over all actions; failed tasks score 0. On 84 labels a human agrees with GPT-4.1 on 82% (Cohen’s κ=0.64). Swapping the judge for Claude Sonnet 4 or Llama-4 Maverick keeps the model ranking and moves absolute CCE by at most about 0.05. The labeling policy is inclusive, so CCE is an upper bound on genuinely useful work.
All four models share prompts and tool schemas.
| Model | Main SR | PSR | CCE | Augmented SR |
| Gemini 3 Flash | 52.0% | 71.5% | 0.320 | 24.0% |
| Claude Haiku 4.5 | 45.0% | 62.7% | 0.294 | 26.0% |
| GPT-5 Mini | 36.0% | 62.6% | 0.206 | 21.0% |
| DeepSeek R1-70B | 20.0% | 43.5% | 0.125 | 10.0% |
Twenty-seven percent of tasks are unsolved by every model. Survival is easiest (50–100% SR); coordination, construction, and trading are hardest. More chat is not more collaboration: GPT-5 Mini sends 44.1 messages and 131.3 actions per task, spends 26% of failed-task actions chatting, and ranks third. DeepSeek waits until round 5 to speak (7.5 messages) and finishes last. Gemini uses 26.4 rounds and 11.0 messages and wins. Conditioned on successes, Claude’s CCE is 0.653, Gemini 0.609, GPT-5 Mini 0.571.
Baselines pin the difficulty: random 5.7%, one agent given every role and item 28.6%, no chat 17.6%, a human shared plan with chat disabled 22.9%, oracle communication 60%. On a 35-task slice, dropping task docs cuts Gemini from 54.3% to 29.4%; cutting the horizon from 50 to 10 rounds lands at 37.1%; scrambling spawns only falls to 51.4%. Of clustered chat errors, 37.7% are stale or redundant, then role mix-ups (16.4%) and factual slips (14.8%). Three Gemini reruns on the 35-task slice all report 54.3% SR.
Few benchmarks pull collaboration apart from motor skill and short dialogue. CCE is more reproducible than a 1–5 “coordination” grade, and is usable as a training or regression signal. For people building multi-agent stacks, 52% success and 0.32 CCE say the bottleneck is shared plans and information over dozens of rounds, not tool-call syntax. PSR sitting 20 points above SR means teams can start and still fail the last handoff.
It is still a game. Whether software or lab work fails the same way is untested. The environment, tasks, verifiers, and annotation UI are open, so the reproduction cost is low.
Failed tasks contribute CCE=0, so the average is tied to success rate; ranking on successes only already moves. The judge is still an LLM, with the question reduced to binary causality. Ablations and the three-seed check use 35 tasks and one model; pairwise model gaps have no significance test. Round-robin turns remove concurrency and the timing noise of a live world. Augmented tasks were written by Claude Opus; bias toward Anthropic models is unmeasured. The four primary models are all small/fast SKUs (Flash, Haiku, Mini, R1-70B). “Foundation models cannot collaborate” does not automatically extend to their flagship counterparts.