Study Proposes Nine Strategies for Selecting Models in Multi-LLM Collaboration, Beating Random by up to 36.1%
In the paper "You're Hired: Strategic Model Selection for LLM Collaboration" (arXiv:2609.38816), Shangbin Feng, Zongwan Cao, and colleagues at the University of Washington study model selection in multi-LLM collaboration systems: given over 2 million open-source models, how should a collaborative team "hire"?
Confirmed
- They propose and systematically evaluate 9 model selection strategies, spanning description-based diversity, item response theory (IRT), and individuality signals; the best strategy beats random selection by up to 36.1%
- Experiments use two candidate pools: 10 specialized Qwen models and 32 models independently contributed by the research community; each team hires 4 models, evaluated via four collaboration algorithms including prompt routing, multi-agent refine, and DARE-T
- Random team formation is a lottery: performance can swing by ±25 points; picking the strongest models by individual capability can backfire: under multi-agent refine, TruthfulQA scored only 24.8, below the random baseline of 45.4
- When 1-6 misaligned/unsafe models with seemingly harmless descriptions are mixed into the candidate pool, Agentic interviewing and Nested selection keep teams 81-83% clean, while description-based screening is insufficient to filter out such models
Why it matters
With open-source models proliferating, how to scientifically staff multi-LLM collaboration systems has become a key new question. The study shows that "stacking the strongest models" is not optimal; capability-aware and interview-style screening strategies both boost performance and defend against infiltrating unsafe models, offering an actionable strategy framework for building multi-LLM systems.
2026-10-23 ~ 2026-10-23 · 5 related posts
Primary sources
- Researchers test 9 strategies to pick LLM teams, beating random selection by up to 36.1% — shangbinfeng ·
- Paper: strategic model selection lifts multi-LLM teams by up to 36.1% over random hiring — shangbinfeng ·
- Picking top solo models for LLM teams backfires: 24.8 vs random's 45.4 on TruthfulQA — shangbinfeng ·
- [source] Researchers test 9 strategies to pick LLM teams, beating random selection by up to 36.1% — shangbinfeng · 2026-10-23
- Test setup: 10 and 32 model pools, four collaboration algorithms for LLM teams — shangbinfeng · 2026-10-23
- [source] Picking top solo models for LLM teams backfires: 24.8 vs random's 45.4 on TruthfulQA — shangbinfeng · 2026-10-23
- Agentic interviews filter misaligned models, keeping LLM teams 81-83% clean — shangbinfeng · 2026-10-23
- [source] Paper: strategic model selection lifts multi-LLM teams by up to 36.1% over random hiring — shangbinfeng · 2026-10-23