New Benchmark for LLM Multi-Agent Collaboration
j_foerst · x · 2026-07-14
This post introduces a new multi-agent benchmark evaluating how 13 modern LLMs collaborate in long-horizon, open-ended worlds to explore, communicate, trade resources, craft tools, build structures, and combat monsters.
Key findings: Most agents performed poorly, averaging only about 6% normalized return. However, under the most difficult settings, zero-shot Gemini 3.1 Pro matched the performance of an optimal MARL agent trained for 1 billion environment steps.
The authors conclude that coordination capability is an independent bottleneck, beyond just "the ability to complete long tasks." Ablation studies showed that communication had the greatest impact on results.
Related event: Studies Highlight Deficiencies in LLM Multi-Agent Collaboration(6 posts)→
More from Models
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27
- ARC Prize redesigns leaderboard: highest reasoning effort by default, CSV export — mhmazur · 2026-10-27
- Cohere: open-weight Qwen3.8-Flash-Next tried uploading probe file to Beijing cloud bucket in sandbox — wstone_bd · 2026-10-27
- Grok Bot auto-connects to X Money accounts without opt-in, user pulls funds — noahsolomon · 2026-10-27
- Model refuses wrong psychiatric advice in deliberately misleading test — FellMentKE · 2026-10-27
- Clinical System One v0.2: top clinical AI drops from 97% to 65% on harder cases — MaziyarPanahi · 2026-10-27