Mixture of Complementary Agents for Robust LLM Ensemble
Yichi Zhang, Kevin Lu, Yuang Zhang, Jie Gao, Lirong Xia, Fang-Yi Yu · 2026
Read Mixture of Complementary Agents for Robust LLM Ensemble by Yichi Zhang, Kevin Lu, Yuang Zhang, Jie Gao, Lirong Xia, Fang-Yi Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24048.txt.
Introduction
Zhang and five co-authors reframe picking bidders for language-model teams as a feature-selection problem, where a model's value lies in backing the others. Current methods stress scores or variety but miss links among bidders and with the editor, and feature-picking rules cost too much time here. The team therefore tests a wide range of feasible greedy-style pickers that judge fit from a small tagged set. Tests confirm fit as a guiding principle and name the methods with the best score-cost trade-offs in practice.
Catalog record
| CREATOR | Yichi Zhang, Kevin Lu, Yuang Zhang, Jie Gao, Lirong Xia, Fang-Yi Yu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67906 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24048
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mixture of Complementary Agents for Robust LLM Ensemble Yichi Zhang, Kevin Lu, Yuang Zhang, Jie Gao, Lirong Xia, Fang-Yi Yu Abstract Multi-AI collaboration—such as ensembling or debating large language models (LLMs)—is a promising paradigm for aggregating information and boosting performance. A foundational step in these pipelines is to feed the responses of several proposer LLMs into a summarizer LLM, which synthesizes a better answer. However, choosing which proposers to include is non-trivial. Existing approaches primarily focus either on accuracy (picking the strongest models) or diversity (ensuring variety), and often overlook the interactions among proposers and with the summarizer. We reframe proposer selection as a combinatorial selection problem akin to feature selection, where the value of an LLM lies in its complementarity with others. However, directly applying standard feature-selection algorithms is impractical in the LLM setting due to prohibitive time complexity. Motivated by this limitation, we explore an extensive range of computationally feasible, greedy-style selection algorithms that assess complementarity using a small labeled set. Our experiments validate complementarity as a guiding principle for proposer selection and identify methods that achieve the best performance–cost trade-offs in practice. 1 Introduction As today’s Large language model (LLM) ecosystem fragments into numerous models with diverse expertise, collaboration among LLMs has become promising and sometimes necessary for tackling emerging tasks such as mathematical reasoning ( 7 ) , code generation ( 23 ) , and complex decision-making ( 40 ) . A convenient instantiation is ensemble after inference , which aggregates the LLM outputs after the generation of full responses. This includes well-studied frameworks such as LLM debating ( 7 ; 9 ; 2 ) , in which multiple models iteratively exchange arguments before a final decision is reached, and mixture-of-agents (MoA) ( 38 ; 18 ) , which uses layered and summarization schemes to combine diverse model outputs. A fundamental step in the ensemble framework is inputting the responses of N LLM-prompt pairs —the proposers 1 1 1 The same LLM under different prompts can be viewed as different proposers. —into an aggregating LLM—the summarizer —which synthesizes a potentially improved answer. Selecting which proposers to include is therefore critical: for a large proposer pool, it is impractical and inefficient to input responses from every available proposer due to context-window limits and the degraded inference ability ( 19 ) . Existing methods often choose a small set of proposers based on their independent performance, primarily following two heuristics: (i) accuracy-seeking —prioritize high-accuracy proposers or even a single top model with multiple samples ( 18 ; 13 ) , and (ii) diversity-seeking —explicitly mix heterogeneous outputs or prompts to avoid reinforcing similar mistakes ( 17 ; 38 ) . However, both heuristics overlook a decisive factor: the complementarity of proposers both with one another and with the summarizer. We argue these team effects, rather than individual quality or pairwise diversity, ultimately determine ensemble performance. In particular, accuracy-seeking methods rank proposers only by their individual performance, while diversity-seeking methods reward variance regardless of quality. We instead propose mixture-of-complementary-agents (complementary-MoA) —a framework that selects proposers for how well they work together as a team and with the summarizer. The importance of complementarity can be observed from Figure 1 , which compares summarizer accuracy when inputting (i) the individually most accurate proposer versus (ii) the proposer that most complements the summarizer. In this example, we consistently observe a nontrivial gap between the two choices, and furthermore, the most-complementary proposer is sometimes weak on its own. The upshot is both a promise and a challenge: as the ensemble size k (the number of proposers selected for input) grows, the optimal selection can yield substantial gains, yet it also complicates the search, since optimal teams cannot be inferred from individual performance alone. Figure 1 : Summarizer accuracies on AIME ( 6 ) when inputting the most accurate proposer vs. the most complementary proposer. For each summarizer s , the proposer pool is {Qwen3-32B, Sky-T1-32B-Preview, Aya-expanse-32B, Gemini-1.5-Pro, Llama-3.3-70B-Instruct, AceReason-Nemotron, and GPT-4o}, excluding s itself. In this paper, we study efficient proposer selection for multi-LLM ensembling given a summarizer, with a focus on selecting complementary proposers rather than merely strong or diverse ones. Although proposer selection bears superficial resemblance to classical problems such as feature selection or data acquisition, the LLM setting introduces fundamentally new challenges: each evaluation of a candidate proposer team requires expensive summarizer calls, so wrapper-style methods are computationally prohibitive at scale. As a result, existing methods cannot be directly applied, and naive adaptations are prohibitively slow or brittle in practice. See Section 2 for more discussions. Using multiple-choice QA as a concrete testbed, we frame proposer selection as a complementarity-driven optimization problem where the goal is to select a small ensemble that maximizes downstream accuracy while minimizing summarizer calls. This formulation captures both cross-model diversity and intra-model prompt variation, which prior work has shown to be a major source of performance gains ( 18 ; 17 ) . Motivated by this perspective, we develop a family of proposer-selection algorithms that navigate the accuracy–efficiency trade-off. We first introduce model-first greedy , a wrapper-style method that keeps the summarizer in the loop but reduces query complexity, measured by the total number of summarizer calls required for proposer selection. We select the winning model at each step based on the average marginal gain its prompt variants provide to the current ensemble S . Once a model is selected, we identify its best-performing prompt instance and add that model-prompt pair to S . Model-first greedy reduces query complexity based on the intuition that diversity across models matters more than diversity induced by different prompts within the same model. To further reduce query complexity, we propose two algorithms that consider label-level complementarity. Truth-prediction greedy selects proposers based on how well their reported labels help predict the ground truth; oracle-surrogate greedy first fits a simple surrogate of the oracle and then selects proposers based on their marginal contributions measured by the surrogate model. Both methods rely only on label-level statistics and therefore require no—or only light—summarizer calls. We conduct an extensive empirical study across three popular reasoning benchmarks, spanning multiple proposer pools (a dominating-LLM regime and a mixed-crowd regime), different summarizers, and a range of ensemble sizes. This evaluation reveals a consistent pattern: commonly used heuristics perform well only in narrow regimes and fail unpredictably elsewhere. In contrast, our complementarity-guided methods, including the truth-prediction greedy which requires no summarizer call, are consistently robust across all scenarios. Moreover, we frequently observe substantial gains from model-first greedy over the strongest baseline, underscoring that explicitly optimizing for complementarity is crucial in ensemble frameworks. In summary, our main contributions are threefold: • We identify complementarity as a key, yet overlooked objective in agent-level LLM ensembles and propose a more principled proposer-selection framework, called complementary-MoA, that explicitly optimizes it. • Inspired by feature-selection methods, we present three complementarity-driven selection algorithms designed around LLM ensembles, where evaluating each proposer requires expensive model calls. The resulting methods span a spectrum of accuracy–efficiency trade-offs, giving practitioners a principled way to choose under different query budgets. • Through large-scale experiments, we provide a systematic and comprehensive comparison of LLM-compatible proposer-selection strategies, revealing the failure modes of existing heuristics and demonstrating that complementarity-based selection delivers the