NVIDIA paper: multi-model ensembles often underperform, single-family selection wins
omarsar0 · x · 2026-09-16
An NVIDIA team studied how to choose models for multi-agent systems, comparing eight selection strategies—by size, accuracy, answer diversity and error diversity—across routing, majority vote and LLM-as-judge setups on hard science benchmarks.
Key findings:
- Larger pools of diverse open models raised theoretical best-case accuracy, but achieved accuracy often fell below the single best model in the pool
- Using several copies of one model worked better
- Majority vote over the best single model lifted HLE accuracy from 29.4% to 32.2%, while nearly every mixed-model group declined
- Choosing candidates from a single model family gave the largest improvement of all eight strategies
Takeaway: before adding another model to a router or ensemble, measure what it adds.
More from Research
- Specialized inference engines: the batch-of-prompts abstraction may be wrong — sh_reya · 2026-09-17
- OmniHarness: symbolic policy learning boosts generalizable visual generation — Xu Xu · 2026-09-17
- TokenRhythm Launches NeoHorse-1: 4B/9B Models Post-Trained on Agent Execution Traces — rohanpaul_ai · 2026-09-17
- Stanford's James Zou Shows Paper MCPs: Agents From Different Papers Collaborating on Research — james_y_zou · 2026-09-17
- Paper2Agent detail: AI builds a virtual lab to reproduce papers — james_y_zou · 2026-09-17
- Paper2Agent's auto-generated paper MCP beats paper + code repo — james_y_zou · 2026-09-17