Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systems
Sara Vera Marjanović, Jiacheng Xu, Aleksandr Laptev, Grigor Nalbandyan, Erik Arakelyan, Evelina Bakhaturina
EMNLP 2026
cs.MA, cs.AI
2026-09-15
NVIDIA tests 23 open models and 8 pool rules on HLE-class science tasks. Oracle bounds rise with pool size; real MAS often lose to the best member except same-family pools.
Multi-agent systems (MAS) usually take one of two shapes. A router picks a model before anyone generates. Or several models generate first, and a majority vote or an LLM judge collapses the answers. Homogeneous setups, same base model with extra samples or prompt variants, often improve as you add agents. Heterogeneous setups, mixing families and sizes, have been unstable for years.
There are millions of models on Hugging Face. How to pick the candidate pool is still under-studied. This NVIDIA paper (EMNLP 2026, with a University of Copenhagen internship) treats pool selection as a design variable, not a footnote.
The pool has 23 open models from 2024 to 2026, 2B to 1.6T parameters: Qwen3/3.5, Gemma 4, OLMo 3, Llama 3.1, gpt-oss, DeepSeek-v4, MiniMax, plus four science-tuned specialists. Each model produces 5 samples per question. Test sets are Humanity's Last Exam (HLE), GPQA-Diamond, and Frontier Science-Olympiad. Open-ended answers are graded by gpt-oss-120b, which matches a human on 93% of 100 HLE items (Cohen's kappa 0.63) and is biased toward marking answers correct.
A 15.5k-question calibration set from AOPS, Turing, Scale, and Stack Overflow tracks relative model ranking on the test sets (r>0.9).
Eight selection rules: size (one model per family), whole families, GPT-5 Deep Research picks, pass@1, Jaccard distance of correct-answer sets, error diversity on multiple-choice items, and two 50/50 mixes of accuracy with diversity. Pool size k is 3, 5, 10, 15, or 20.
Three MAS:
The headline metric is ΔMAS Gain versus the best single model already in that pool.
Accurate models get the same questions right (Mantel rM=0.931). Their errors only weakly overlap (rM=0.384). Size correlates moderately with accuracy (Spearman rs=0.583). Vanilla Llama-3.1-8B beats its physics- and chemistry-tuned cousins on those very domains. Specialist fine-tunes do not create the complementary expertise a router would want.
Oracle upper bounds rise with k. Accuracy-based and GPT-5-chosen pools look best on paper. Real systems invert the plot: larger k usually hurts relative to the best member. On HLE, only same-family pools beat that baseline. Gemma 4 is the most consistent family; OLMo 3 and Qwen3 swing around.
Homogeneous sampling still helps. Heterogeneous majority vote and judging drop faster than routing as k grows. Only IoU-diverse and family pools show occasional positive routing gains. On HLE, intentional judge pools often lose to random pools; on GPQA and FS they roughly match random. Accuracy-selected pools pick up about +2 points on FS and under +1 on GPQA.
| Setting | Metric | Number |
| Best single model, HLE | pass@1 | 29.4% |
| Homogeneous majority@5 | HLE | 32.2% |
| Homogeneous judge@5 | HLE | 36.5% |
| Heterogeneous larger k | vs best member | mostly negative |
| Judge vs human | agreement | 93% (kappa 0.63) |
Do not dump popular open models into one MAS. Same family, similar strength, consistent output format beat "more and more diverse." Routers need learnable expertise gaps, not a crowd of generalists that solve the same items. The study uses simple MAS, so tool-using or debate-style systems might behave differently. Pool choice still belongs in the ablation. "More agents" is not a default.
Homogeneous systems gained as the architecture got fancier. Heterogeneous systems lost. Voting and judging look built for same-model samples. Mixed families may need a different aggregator.
The authors stay with before- and after-generation MAS and skip during-generation collaboration. One router backbone. Context limits may hurt GenSelect at large k. Tools and retrieval are off, which is a real handicap on science questions. Family groups cannot hit every k. Science reasoning only. The judge is lenient, so absolute scores are inflated.
The main text mostly shows curves, not absolute accuracy tables per strategy. That makes the paper hard to quote as an engineering lookup table. Disabling tools may also exaggerate how badly answer diversity pollutes majority vote.