FULL STORY

Test-Time Communication: 5 Cooperating Agents Match 33

A new paper from UC Berkeley and Microsoft Research claims 5 cooperating agents can match 33 independent ones, but follow-up experiments showing a 10-100x compute cost ignited debate over multi-agent versus serial baselines.

2026-09-21 ~ 2026-09-23 · 2 episodes · 33 posts

Episode 1 · Test-Time Communication: Team-of-5 Agents Matches Best-of-33 (2026-09-21, 25 posts)

Starting September 21, Dimitris Papailiopoulos and colleagues (UC Berkeley and Microsoft Research; first author Jongho Park, with Akshay Krishnamurthy among the authors) released "Scaling Discovery through Test-Time Communication," proposing test-time communication as a possible next scaling axis beyond inference scaling. The central question: when does Team-of-N beat Best-of-N? The answer: whenever agents can communicate. The work began months before the recent Hugging Face incident, in which agents fully exploited any communication channel they found—an inspiration and an extreme example of collaborative behavior.

Confirmed

  • Setup: N identical agents tackle the same task with no preset roles, coordinating only through a shared plain-text log and simply told to "collaborate."
  • Key data (relayed by ARC Prize's Greg Kamradt): on ARC-AGI-3, a team of 5 Claude (sonnet-4.6) agents matches Best-of-33, solving 65% of tasks; other accounts state the collaborating team performs as well as roughly 4x as many independent agents.
  • Across three research-style task families, communicating teams substantially outperform independent agents; the authors interpret communication as shifting agents from parallel search to cumulative discovery, letting teams solve problems no single agent could.
  • First author Jongho shared a research thread on whether agents can collaborate effectively and whether more agents help, citing OpenAI's prior work on swarms of 10,000 collaborating agents. Jon Ghoh and Dimitris Papail noted many signals of gains from multi-agent communication existed but lacked systematic quantification, which this paper aims to fill.

Reactions

  • Sarah Hooker commented that compute is shifting from pretraining (diminishing FLOPs returns) to test-time compute, requiring very different infrastructure; the first agent-designed infra wave is already landing.
  • mariofilhoml called the work very cool, drawing an analogy to a paper titled "A PROOF OF THE CYCLE DOUBLE COVER CONJECTURE" as a multi-agent collaboration pattern.
  • gleech argued AI progress may be faster than expected because many breakthroughs look "a bit silly"—agents communicating beating independent best-of-N is a counterintuitive yet simple finding.

Why it matters

  • If test-time communication becomes a real scaling axis, capability gains shift from stacking pretraining compute toward multi-agent collaboration architectures, spurring a new generation of agent-oriented infrastructure.

5 more related posts →

Episode 2 · Team-of-N Agents Beats Single Agent by 10-100x Tokens, Sparking Baseline Debate (2026-09-22, 8 posts)

On September 22, Dimitris Papailiopoulos, author of a multi-agent collaboration paper, kicked off a debate that became the day's focal discussion on agent scaling. On MNIST, his team-of-N vs. single-agent curves suggest a single best agent needs 10-100x the token budget to match the team; ablations further show that on some problems N agents running in parallel for 1x time beat one agent running for Nx time — making multi-agent compute-optimal, not just faster. generatormanai then challenged the methodology, and several researchers added experiments and explanations.

Confirmed

  • DimitrisPapail showed MNIST team-of-N vs. single-agent best-of-N curves, inferring a single agent needs 10-100x tokens to match team performance.
  • Ablation data he shared shows that on some problems, N agents in parallel for 1x time beat one agent running Nx time — parallel agents are compute-optimal for some problems.
  • He added that in another project, periodically clearing an agent's context while keeping only a summary document of "what happened so far" also beat the best single agent.
  • jonghoh cited ARC-AGI-3 ablations: with sufficient compute, single agents with larger token budgets plateau faster than same-budget best-of-N, meaning team-style multi-agent scales better under equal total compute.
  • ShangyinT summarized converging observations: best-of-N@T under equal compute far outperforms serial best-of-1.
  • DimitrisPapail offered a cautious mechanism: models become "narrow-sighted" on a single very long context (NT), while injecting entropy across multiple API calls may help — stressing this is an unscientific hunch.

Unconfirmed

  • generatormanai's methodological objection remains unresolved: the correct serial baseline for team-of-N at budget T should be a single agent with budget NT (best-of-1@NT), not best-of-N; no correction appears in the material.
  • No evidence yet on whether the MNIST findings generalize to complex real-world tasks.

Why it matters

  • If the 10-100x efficiency advantage and parallel compute-optimality hold broadly, it directly shapes agent architecture choices: prefer teams over giant single contexts under equal compute.
  • Periodic context clearing plus a summary document is a cheap, immediately applicable engineering trick aligned with long-context attention degradation.
  • The baseline dispute warns the community: multi-agent papers lacking a best-of-1@NT baseline may overstate efficiency gains.