When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines
Artem Maryanskyy · 2026
Read When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines by Artem Maryanskyy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20324.txt.
Introduction
Maryanskyy and co-authors study why multi-agent language model teams give mixed results. Diverse teams sometimes beat single models, yet uniform teams win under synthesis. They name the cause the selection bottleneck, a crossover point in aggregation quality. Past that point, diversity helps; below it, diversity hurts. In tests on 42 tasks, a diverse team with judge selection won 81% of comparisons. Synthesis-based merging won none of the 42 tasks. Selector quality, they argue, matters more than team variety.
Catalog record
| CREATOR | Artem Maryanskyy |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51024 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20324
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines Artem Maryanskyy Abstract Multi-agent LLM pipelines produce contradictory evidence on whether team diversity improves output quality: heterogeneous Mixture-of-Agents teams outperform single models, yet homogeneous Self-MoA teams consistently win under synthesis-based aggregation. We propose a resolution by identifying the selection bottleneck —a crossover threshold in aggregation quality that determines whether diversity helps or hurts. Under this model, we obtain a closed-form crossover threshold s ∗ s^{*} (Proposition 1 ) that separates the regimes where diversity helps and hurts. In a targeted experiment spanning 42 tasks across 7 categories ( N = 210 N=210 ), a diverse team with judge-based selection achieves a win rate of 0.810 against a single-model baseline, while a homogeneous team scores 0.512—near chance (Glass’s Δ = 2.07 \Delta=2.07 ). Judge-based selection outperforms MoA-style synthesis by Δ WR = + 0.631 \Delta_{\mathrm{WR}}=+0.631 —the synthesis approach is preferred over the baseline in zero of 42 tasks by the judge panel. A decoupled evaluation with independent judges confirms all directional findings (Spearman ρ = 0.90 \rho=0.90 ). Exploratory evidence suggests that including a weaker model improves performance while reducing cost ( p < 10 − 4 p<10^{-4} , not pre-registered). Our results suggest that selector quality may be a more impactful design lever than generator diversity in single-round generate-then-select pipelines. 1 Introduction “A group of diverse problem solvers can outperform a group of high-ability problem solvers”—so concluded Hong and Page [ 8 ] in one of the most cited results in collective intelligence. Two decades later, the multi-agent LLM community is reliably reproducing both sides of this claim. Wang et al. [ 22 ] demonstrated that mixing models from different families into a Mixture-of-Agents architecture improves response quality over any single model, declaring their approach “achieve[s] state-of-the-art on AlpacaEval 2.0, MT-Bench, and FLASK.” Li et al. [ 14 ] tested the same premise under controlled conditions and reached the opposite conclusion: Self-MoA—a homogeneous team of identical models—“consistently outperforms Mixed-MoA across all benchmarks.” Both papers are methodologically sound. Both claim generality, yet their conclusions appear contradictory. We identify a mechanism that reconciles this tension: the selection bottleneck . Whether a diverse team’s high-variance candidate pool is an asset or a liability depends entirely on how those candidates are aggregated. A synthesis-based aggregator—the method both prior works employed—compresses all candidates into one blended response, forfeiting the advantage of having generated a standout candidate. A selection-based aggregator evaluates candidates individually and picks the best, exploiting exactly the variance that synthesis wastes. The distinction is not a nuance; it is the dominant factor determining whether diversity helps. In a targeted experiment crossing team composition with aggregation mechanism across 42 tasks spanning seven categories, we observe a crossover. A diverse team dramatically outperforms a homogeneous team under judge-based selection, but the same diverse team provides no advantage under majority voting. The largest effect in our data is the comparison practitioners most need: judge-based selection versus MoA-style synthesis. Selection wins in every single one of the 42 tasks across all 7 categories. The synthesis approach, far from combining the best of multiple agents, produces outputs that lose to a single-model baseline over 80% of the time. The apparent paradox may dissolve: Li et al. and Wang et al. each describe one side of a crossover governed by aggregator quality. A simple linear model makes this precise. We define a selector quality parameter s\in[0,1] and obtain a crossover threshold s^{*} below which diversity hurts and above which it helps (Proposition 1 ). The model predicts that selector quality should have no effect for homogeneous teams, consistent with the observed near-chance performance of the homogeneous cell in our data. When all candidates are drawn from the same distribution, there is nothing for even a perfect selector to exploit. Figure 1 illustrates the crossover geometry. One finding warrants particular attention, though we label it exploratory because it was not pre-registered. Adding a substantially weaker model (Claude Haiku) to a strong diverse team is associated with a significantly higher win rate and lower cost. The mechanism is plausible: a model from a different capability tier introduces orthogonal error patterns, raising the oracle ceiling even as it lowers the team mean. A strong selector captures the upside and ignores the downside. This paper makes three contributions. First, we introduce the selection bottleneck model , which provides a crossover threshold s^{*} unifying pro-diversity and anti-diversity findings (Proposition 1 ). The threshold offers one operationalization of Hong and Page’s abstract notion of an “effective aggregation mechanism.” Second, we present empirical evidence that selection dramatically outperforms synthesis —the aggregation mechanism used by both Wang et al. and Li et al.—with the largest effect size in our study, demonstrating that the MoA paradigm of blending candidates is substantially outperformed by selection in our setting. Third, we report exploratory evidence of a weak-model paradox : including a cheaper, weaker model may simultaneously improve quality and reduce cost, a finding that, if replicated, inverts conventional wisdom about team composition. Figure 1: The selection bottleneck. (a) Theoretical output quality Q(T,s) as a function of selector quality s . The diverse team (blue) crosses the homogeneous baseline (gray) at the crossover threshold s^{*} (dashed vertical). Below s^{*} , diversity hurts; above it, diversity helps. (b) Empirical operating regimes from V4 data. Judge-based selection operates well above s^{*} (WR = 0.810), majority vote sits near s^{*} (WR = 0.496), and MoA-style synthesis falls far below it (WR = 0.179). The homogeneous baseline (WR = 0.512, dotted) is shown for reference. 2 Related Work Multi-agent architectures for large language models have moved from curiosity to engineering pattern in under two years, yet the field lacks a unifying account of when combining models helps and when it does not. We organize prior work into five threads and identify the gap our framework addresses. 2.1 Multi-Agent Debate and Frameworks Multi-agent frameworks such as AutoGen [ 24 ] , CAMEL [ 12 ] , and MetaGPT [ 9 ] enable flexible orchestration topologies where agents converse and collaborate. Du et al. [ 7 ] showed that multi-agent debate improves factual accuracy by forcing agents to defend claims against challenges. Smit et al. [ 16 ] found substantial variation across task types, and Choi et al. [ 5 ] uncovered identity bias —LLMs preferentially agree with their own prior outputs—undermining the independence assumption. Identity bias is directly relevant to our framework: it manifests as elevated inter-agent correlation \rho , reducing the effective sample size of any aggregation mechanism that relies on independence. 2.2 Mixture-of-Agents and Self-MoA Wang et al. [ 22 ] proposed Mixture-of-Agents (MoA), a layered architecture in which diverse “proposer” models generate candidates that an “aggregator” synthesizes into a final output. MoA with heterogeneous proposers outperformed any single model. Li et al. [ 14 ] challenged this with Self-MoA, reporting that homogeneous teams “consistently outperform Mixed-MoA.” Both analyses share a critical blind spot: neither varied the aggregation mechanism. Both used synthesis-based aggregation, operating at a fixed selector quality s that our framework identifies as the key moderator. The contradiction is not about diversity per se but about whether the aggregator operates above or below the crossover threshold s^{*} . 2.3 Selection and Routing