Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems
Jinxi Yu, Yubei Li, Eric Hanchen Jiang, Zhi Zhang, Dong Liu, Wenxiao Zhao, Levina Li, Kai-Wei Chang, Ying Nian Wu
cs.AI, cs.LG, cs.MA
2026-09-02
A 16-entry VQ codebook plus an MLP proxy selects multi-agent topologies in 2.4 ms, averaging 84.6 accuracy versus 83.0 for the best prior designer, with 21.9–33.2% fewer tokens.
Which agents talk to which, and in what order, moves both accuracy and token spend in an LLM multi-agent team. A fixed graph is cheap to deploy and a poor fit for every task: dense connectivity can help math, a star can help code. Recent designers therefore emit a fresh topology per query. A variational, autoregressive, or diffusion decoder searches the N×N adjacency, and a graph network ranks candidates on utility plus a structural cost, usually the edge count.
Three measurements in this UCLA paper say that recipe is misaligned. Topologies that survive a reward filter collapse to about six distinct graphs even as codebook capacity grows from 8 to 64, and the best fixed graph stays within 1.4 accuracy points of every generated one. Edge count correlates with measured tokens at Pearson r≈−0.4: sparser graphs produce longer completions, so minimizing |E| raises the bill it was meant to cut. On the homogeneous teams that dominate published benchmarks, a message-passing scorer over profile nodes is invariant to adjacency. Every candidate gets the same score.
Codebook Agent amortizes design into three feed-forward pieces and drops the search loop at test time.
Offline, each benchmark runs 50 training tasks under 6 fixed topologies (complete, chain, star, three Erdős–Rényi samples), logging 300 tuples of adjacency, query embedding, binary utility, and token count. The composite reward is utility minus λ=0.1 times tokens normalized by the same-task mean, so difficulty cancels.
At test time the predictor’s top-5 codes are decoded, deduplicated, and scored in one batched proxy call. The graph maximizing û−λĉ is executed. Topology generation makes no LLM calls.
On gpt-4o-mini across six benchmarks, Codebook Agent averages 84.62 and leads every column. Vanilla sits at 77.63; the strongest prior designer, GTD, at 83.02. Per-task: GSM8K 94.8, MATH 56.5, MultiArith 99.4, SVAMP 95.4, MBPP 83.5, HumanEval 78.1. Gains over GTD are spread: +1.3 GSM8K, +1.0 MATH, +3.1 MBPP, +0.6 HumanEval.
| Method | Mean accuracy | Topology latency |
| Vanilla | 77.63 | no designer |
| DyLAN | 80.18 | no designer |
| GTD | 83.02 | 301–396 ms |
| Codebook Agent | 84.62 | 2.4 ms |
Tokens drop 21.9–33.2% versus the full incumbent pipeline (iterative decoder plus edge-count GNN): GSM8K 1239→927, MATH 2304→1611, homogeneous HumanEval 699→546, heterogeneous HumanEval 624→417. MATH wall-clock falls from 43.7 to 27.4 minutes. Swapping the backbone to Qwen-3-8B, the method still leads at 74.0 versus GTD 72.7.
Ablations isolate the claims. For every K≥8 the encoder uses at most six codes; extra capacity stays idle. Holding the candidate set fixed, GNN rerank costs 1711 tokens on GSM8K against 1249 for random and 927 for the MLP, with accuracy within 1.5 points. Ranking buys cost, not Pass@1. The best fixed family already sits close to generated graphs.
The field treated topology design as conditional graph generation. This paper treats it as indexing a short list and picking by measured tokens. 2.4 ms is noise next to end-to-end latency; the saving that matters is LLM calls. Anyone still sparsifying graphs to save money should look at the inverted surrogate: an edge-count head pushes the system toward more expensive runs.
The accuracy lift over GTD is about 1.6 points. That is incremental. The contribution is the diagnosis and the amortization, not a stronger generator. A new setting still owes a one-time collection of 300 real executions.
The best fixed topology already hugs the generated numbers, so a large share of the designer’s job is selection among a short list. Each configuration is a single evaluation run, with no variance reported. On heterogeneous HumanEval the incumbent GNN scores 78.8 accuracy against 78.1 here. The negative edge–token correlation comes from this paper’s own logs and may not travel. The codebook is query-independent after training, so a new team size needs a retrain. Transfer to a fully new set of roles is untested.