Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial

Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin · 2025

Read Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial by Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00674.txt.

Description

Mixing answers from several different large language models is a popular way to boost performance, a method called Mixture-of-Agents. The four authors ask whether the mixing itself helps. They propose Self-MoA, which ensembles outputs from only the single best model, and test it widely. Surprisingly, Self-MoA beat the mixed version in many scenarios.

It improved on Mixture-of-Agents by 6.6 percent on AlpacaEval 2.0 and by 3.8 percent on average across benchmarks including MMLU, CRUX, and MATH. Applied to a top leaderboard model, it set a new state of the art. Probing the diversity-quality trade-off, the authors find performance is sensitive to quality, and mixing weaker models drags the average down. They also map when mixing does help, and they add a sequential version that aggregates many outputs over rounds.

Catalog record

CREATORWenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-00674

Browse the full library · About the Tower of Babel