Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial
Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin · 2025
Read Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial by Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00674.txt.
Description
Mixing answers from several different large language models is a popular way to boost performance, a method called Mixture-of-Agents. The four authors ask whether the mixing itself helps. They propose Self-MoA, which ensembles outputs from only the single best model, and test it widely. Surprisingly, Self-MoA beat the mixed version in many scenarios.
It improved on Mixture-of-Agents by 6.6 percent on AlpacaEval 2.0 and by 3.8 percent on average across benchmarks including MMLU, CRUX, and MATH. Applied to a top leaderboard model, it set a new state of the art. Probing the diversity-quality trade-off, the authors find performance is sensitive to quality, and mixing weaker models drags the average down. They also map when mixing does help, and they add a sequential version that aggregates many outputs over rounds.
Catalog record
| CREATOR | Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00674