Eight coordinates to describe any multi-agent research system, and why 'lack of taste' is two different failures

A Vocabulary for Multi-Agent Automated Research Systems

Bardiya Akhbari

cs.AI, cs.LG, cs.MA

2026-07-13

An eight-coordinate vocabulary for describing and comparing multi-agent automated research systems isolates each design choice, and splits the vague 'lack of taste' into generative and evaluative failures.

What problem this solves

A wave of multi-agent automated research systems has arrived: AI Scientist, AIRA, AlphaEvolve, Glia, each claiming to run research end to end. The trouble is they differ along several axes at once: how agents communicate, whether they remember across runs, how they are initialized, how outputs are scored. So the claim that multi-agent beats single-agent is meaningless, because you cannot tell which axis drove the gain. The author wants a shared language that isolates design choices so they can be compared.

Method

The vocabulary is a tuple. A system is ℳ = ⟨A, O, C, α, S, π, ι, e⟩:

A (agents): each agent records backbone model, private memory, initial memory, role.
O (operation universe): callable tools, skills, APIs, shell, that is, what can be done.
C (communication): who may message whom (Λ) plus the message schema (σ).
α (capability assignment): which agent may invoke which operation, separating what exists from who may use it.
S (shared state): within-run scratchpad (Sbtw), external world state (Sworld), and cross-run state such as skill libraries or population databases (Scross).
π (control): routing (who acts next), stopping, meta-control (spawn/kill, fork/join, grant/revoke mid-run), and exploration η.
ι (initialization): starting memories, state, capabilities.
e (evaluator): the proxy scorer, with integrity protections against reward hacking.

The load-bearing choice is making the evaluator a component of the system, because reported gains depend on how closely the proxy matches true quality.

Two sharp concepts:

Overfitting tax: the more trajectories a search ranks, the more likely the top one is a case where the proxy score overstates true quality. Ranking is itself a source of overfitting.
Taste split: saying a system has no taste is vague. Split it into generative taste (rate of proposing novel trajectories before any score) and evaluative taste (gap between proxy score and true quality). Two failures, two fixes.

A trajectory records one run from input to returned artifact. Because of stochasticity, repeated runs on the same task induce a distribution over trajectories, not a single behavior.

The author then maps 10 real systems (AIRA2, AlphaEvolve, Glia, AAR, AI Scientist-v2, MetaGPT, EvoX, ml-intern, SimpleTES, Engram) onto the vocabulary to see which coordinates each one actually sets. For example, Glia's Supervisor holds no operations (α(Sup) = ∅), AI Scientist-v2 sets mainly meta-control with a fork-prune tree search and Scross empty, and MetaGPT is a fixed handoff chain.

Results

There are no quantitative experiments; this is a conceptual contribution. The result is a coverage analysis: each of the 10 systems sets only a few coordinates and leaves the rest at default. A few observations:

No analyzed system changes the capability assignment α during a run (blocked by the engineering cost of permission infrastructure).
η (exploration/stochasticity) stays at default in every system.
Architecture-task alignment predicts gains better than agent count |A|. Adding agents is not the lever; matching structure to task is.

To be blunt: there is no benchmark and no SOTA number. The output is a map, not a measurement.

Why it matters

For anyone building or reviewing agentic systems: instead of asking whether multi-agent is better, ask which coordinate changed. That matters because the field is drowning in systems that vary everything at once and report gains you cannot attribute. The overfitting tax is the most actionable warning: when a system ranks many candidate runs by a proxy scorer, the winner is biased toward proxy-overstated quality, so a large search budget can backfire if the proxy is miscalibrated. The taste split tells you whether to fix the generator (propose more diverse candidates) or the evaluator (tighten proxy to quality).

Limitations

The author concedes several coordinates are under-exercised: no system varies α mid-run, η is always default, the task distribution is fixed, so the vocabulary's discriminating power there is untested.
Generative taste stays qualitative: there is no novelty measure to score the generator independently.
True quality q is generally uncomputable; you can design toward it but not measure it, so evaluative taste can only ever be approximated, never quantified directly.
One more layer: as a position paper it lives or dies by adoption. Whether these eight coordinates become the shared language or get superseded is unresolved. Single author, no experiments means this is a proposal, not a validation.

Terms

trajectory: one run recorded from the input task to the returned artifact.
Scross (cross-run state): anything persisted across runs, such as a skill library or an evolutionary population database; setting it empty means the system starts from scratch each time.
capability assignment α: which agent may invoke which operation, separating what operations exist from who may use them.
meta-control πmeta: changing system structure mid-run, such as spawning or killing agents, forking or joining, granting or revoking capabilities.
overfitting tax: when an evaluator ranks candidates, the top one is increasingly likely to be one where the score is inflated.
generative taste / evaluative taste: generative taste is whether the system proposes novel trajectories before scoring; evaluative taste is the gap between proxy score and true quality.

Source

Related papers

All paper explainers