Mixture-of-Agents Enhances Large Language Model Capabilities
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou
cs.CL
2024-06-07
Mixture-of-Agents stacks LLMs in layers where each agent refines all previous-layer outputs. An open-source-only stack scores 65.1% on AlpacaEval 2.0, beating GPT-4o's 57.5%.
Open-source LLMs keep multiplying, each with its own strengths: some follow instructions well, some write better code, but none dominates every query. The question: without retraining anything, can you merely "team up" these models into a system stronger than any single one?
Mixture-of-Agents (MoA) stacks models in layers, usually three. Each layer holds several LLM agents:
The crux is aggregation. A designed prompt asks the model to merge the reference answers into one, critically sifting the information and dropping errors rather than concatenating.
Models play two roles: proposers, good at producing useful reference answers, and aggregators, good at fusing several answers into one high-quality output. The authors empirically assign roles. The default open-source stack uses Qwen1.5-110B-Chat (final-layer aggregator), Qwen1.5-72B-Chat, WizardLM-8x22B, LLaMA-3-70B-Instruct, Mixtral-8x22B, and dbrx-instruct, reused across three layers.
A cheaper MoA-Lite uses two layers, since the first aggregation gives the largest gain.
| Config | AlpacaEval 2.0 LC win rate |
| GPT-4o (0513) | 57.5% |
| GPT-4 Turbo (0409) | 55.0% |
| MoA (open-source only) | 65.1% |
| MoA + GPT-4o final aggregator | 65.7% |
| MoA-Lite (two layers) | 59.3% |
The open-source-only stack beats GPT-4o by 7.6 points on AlpacaEval 2.0. On MT-Bench, MoA with a GPT-4o aggregator scores 9.40, ahead of GPT-4 Turbo's 9.31. On FLASK dimensions, MoA beats GPT-4o on correctness, factuality, insight, and completeness, at the cost of longer, less concise answers.
On cost, MoA-Lite reaches higher quality at GPT-4o-comparable cost and is about twice as cost-effective as GPT-4 Turbo; multiple proposers run in parallel, cutting latency.
This is a clean demonstration of the "don't train, just ensemble" route: on a strong benchmark like AlpacaEval 2.0, several then-SOTA open-source models stacked together can beat GPT-4o head-on. For teams with limited budgets who don't want to be locked into one closed model, this path was attractive at the time. The cost is upfront: you run several large models at once, so engineering and compute overhead are real.
The biggest pain is Time To First Token (TTFT). The system cannot emit the first token until the last layer finishes, since the final answer depends on all prior layers, which hurts interactive feel. The authors concede this; their mitigations are only "fewer layers" and a hypothetical "chunk-wise aggregation" that they did not implement.
The results lean heavily on a single benchmark, AlpacaEval 2.0, which itself has verbosity bias and is gameable (the LC win rate was introduced precisely to curb gaming). MoA tends to produce longer answers, which lines up with that bias, and the paper does not cleanly separate how much of the win rate comes from "longer" versus "better."
The model lineup is mid-2024 open-source SOTA; much of it has since been surpassed. The method transfers, but the specific numbers are no longer best-in-class.