New Benchmark for LLM Multi-Agent Collaboration
j_foerst · x · 2026-07-14
This post introduces a new multi-agent benchmark evaluating how 13 modern LLMs collaborate in long-horizon, open-ended worlds to explore, communicate, trade resources, craft tools, build structures, and combat monsters.
Key findings: Most agents performed poorly, averaging only about 6% normalized return. However, under the most difficult settings, zero-shot Gemini 3.1 Pro matched the performance of an optimal MARL agent trained for 1 billion environment steps.
The authors conclude that coordination capability is an independent bottleneck, beyond just "the ability to complete long tasks." Ablation studies showed that communication had the greatest impact on results.
Related event: Studies Highlight Deficiencies in LLM Multi-Agent Collaboration(6 posts)→
More from Models
- Google launches three new Gemini models, including a cybersecurity system — Polymarket · 2026-07-22
- Google says information agents are coming to AI Pro and Ultra this summer — gaganghotra_ · 2026-07-22
- Poolside’s Laguna S 2.1 gets a two-week free run on Nous Portal — NousResearch · 2026-07-22
- Qwen3.8 Max Preview looks substantially better in a side-by-side test with Kimi K3 — curiousily_ · 2026-07-22
- Moonshot’s Kimi K3 reaches #5 on MathArena as the top open model — xeophon · 2026-07-22
- Google launches Gemini 3.5 Flash Cyber for CodeMender, with limited access for governments — GoogleAI · 2026-07-22