The Avengers : A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants
Yiqun Zhang, Hao Li, Chenxu Wang, Linyao Chen, Qiaosheng Zhang, Peng Ye, Shi Feng, Daling Wang, Zhen Wang, Xinrun Wang, Jia Xu, Lei Bai, Wanli Ouyang, Shuyue Hu · 2025
Read The Avengers : A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants by Yiqun Zhang, Hao Li, Chenxu Wang, Linyao Chen, Qiaosheng Zhang, Peng Ye, Shi Feng, Daling Wang, Zhen Wang, Xinrun Wang, Jia Xu, Lei Bai, Wanli Ouyang, Shuyue Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19797.txt.
Introduction
Yiqun Zhang and his fourteen co-authors, publishing in 2025, ask whether teams of small open language models can beat the giant closed ones. Their recipe, called the Avengers, sorts each incoming question by topic. It looks up which small models have done best on that kind of question, and lets the top ones answer several times and vote. The system pools ten open models of modest size and needs no retraining.
Across 15 test sets covering math, coding, logic, general knowledge, and emotion tasks, the combined system beat GPT-4o and its newer siblings on average. The margin was largest in math, about 18 points over GPT-4.1, and about 7 points in coding. The team also reports the method holds up on new kinds of questions and does not depend much on which sorting or voting parts are swapped in.
Catalog record
| CREATOR | Yiqun Zhang, Hao Li, Chenxu Wang, Linyao Chen, Qiaosheng Zhang, Peng Ye, Shi Feng, Daling Wang, Zhen Wang, Xinrun Wang, Jia Xu, Lei Bai, Wanli Ouyang, Shuyue Hu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19797
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Avengers : A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants Yiqun Zhang, Hao Li, Chenxu Wang, Linyao Chen, Qiaosheng Zhang, Peng Ye, Shi Feng, Daling Wang, Zhen Wang, Xinrun Wang, Jia Xu, Lei Bai, Wanli Ouyang, Shuyue Hu Abstract Proprietary giants are increasingly dominating the race for ever-larger language models. Can open-source, smaller models remain competitive across a broad range of tasks? In this paper, we present the Avengers —a simple recipe that leverages the collective intelligence of these smaller models. The Avengers builds upon four lightweight operations: (i) embedding : encode queries using a text embedding model; (ii) clustering : group queries based on their semantic similarity; (iii) scoring : scores each model’s performance within each cluster; and (iv) voting : improve outputs via repeated sampling and voting. At inference time, each query is embedded and assigned to its nearest cluster. The top-performing model(s) within that cluster are selected to generate the response with repeated sampling. Remarkably, with 10 open-source models ( ∼ \sim 7B parameters each), the Avengers surpasses GPT-4o, 4.1, and 4.5 in average performance across 15 diverse datasets spanning mathematics, coding, logical reasoning, general knowledge, and affective tasks. In particular, it surpasses GPT-4.1 on mathematics tasks by 18.21% and on code tasks by 7.46%. Furthermore, the Avengers delivers superior out-of-distribution generalization, and remains robust across various embedding models, clustering algorithms, ensemble strategies, and values of its sole parameter—the number of clusters. We have open-sourced the code on GitHub: https://github.com/ZhangYiqun018/Avengers Figure 1: Comparison of proprietary models with our method (the Avengers ) across six representative benchmarks. Full results on all 15 benchmarks are reported in Table 2 . † † footnotetext: Corresponding author. $\clubsuit$ $\clubsuit$ footnotetext: Equal contribution. $\ddagger$ $\ddagger$ footnotetext: Work done during the author’s internship at Shanghai Artificial Intelligence Laboratory. 1 Introduction The current landscape of language models (LMs) is dominated by few proprietary giants. The race to build ever-larger models is far from over; however, it is becoming a closed competition, accessible only to organizations with massive resources. This trend risks marginalizing the broader research community and concentrating innovation within a privileged few. In this context, the open-source community—typically working with smaller LMs—must ask whether it is able to remain competitive. Although recent studies have shown that individual smaller models can excel in specific domains [ 1 , 2 , 3 , 4 ] , the more pressing challenge is: can it remain competitive not only on narrowly defined tasks, but also in achieving comparable or even superior overall performance across a broad range of tasks? A positive answer would have important implications. First, since open-source, smaller models are easily accessible and significantly more amenable, a positive answer would represent a meaningful step toward democratizing AI development [ 5 ] ---enabling a broader community of researchers and practitioners to engage in and contribute to cutting-edge LM research. Second, the open-source ecosystem has already invested significant effort and computational resources in model development. 1 1 1 For example, as of May 2025, there are over 1,318 models fine-tuned on LLaMA-3.1-8B-Instruct and 489 on Qwen-2.5-7B-Instruct, despite both being released less than a year ago. Demonstrating strong performance from these models would justify the reuse of existing efforts, supporting more sustainable AI research [ 6 ] . Third, a positive answer would pave a new path for advancing LM research, providing another viable alternative to the ever increasing of model size. We posit that the key to addressing the challenge lies in collective intelligence —while a single smaller model may not match the generality of larger ones, the collective performance of a team of multiple smaller models may prove competitive. Nevertheless, operationalizing collective intelligence is non-trivial and involves several important design considerations. Since it is infeasible to incorporate all available models in the community, which subset of models should be selected? How should we orchestrate this subset of models to maximize their collective performance? For a notable example, for each incoming query, should we rely on a router-based approach [ 7 , 8 , 9 , 10 ] that selects the most capable model, or should we adopt a mixture-based strategy (e.g., Mixture-of-Agents [ 11 , 12 , 13 , 14 ] ) where multiple models respond in parallel and their outputs are aggregated into a final answer? Moreover, given the rapid evolution of the LM landscape, with new tasks and new models continuously emerging, how can such systems adapt to these ongoing changes? Figure 2: The Avengers , a simple recipe for collective intelligence of smaller language models. To this end, we introduce the Avengers —a simple recipe that effectively leverages the collective intelligence of smaller LMs. Given a set of models and a dataset a dataset (potentially spanning multiple tasks and benchmarks), the Avengers operates as follows. First, it analyzes the tasks by embedding queries from the validation set and clustering them based on their semantic representations. Next, to identify each model’s strengths and weaknesses, it evaluates each model on the validation set and constructs a cluster-wise capability profile—a simple, numeric vector representing model performance across clusters. At inference time, for each query in the test set, the Avengers computes its embedding and assigns it to the nearest cluster. Based on the capability profiles, the model(s) with the highest performance in that cluster are then selected to handle the query. The final output is generated using a repeated sampling and voting strategy over the selected model(s). An overview of the Avengers is presented in Figure 2 . The Avengers is straightforward to implement, requiring no neural network training. It is highly reproducible, avoiding reliance on hand-crafted prompts or architecture-specific model choices. It is plug-and-play, allowing seamless integration of newly available models by incrementally evaluating them on the validation set and computing their capability profiles. Moreover, it is well-suited for cold-start, assuming no human expertise or prior knowledge for model set composition; given a large candidate model pool, the construction of capability profiles enables automatic selection of high-performing models while preserving diversity. Conceptually, the Avengers can be viewed as a routing-based approach. However, as detailed in the Related Work section and summarized in the qualitative comparison in Table 1, it differs significantly from prior work in this category—both methodologically, through its clustering-based routing strategy, and empirically, in the notably strong performance it achieves. Our experiments show that, the Avengers , using 10 models ( \sim 7B each), surpasses the average performance of GPT-4.1 2 2 2 GPT-4.1 : OpenAI’s flagship model released on April 14, 2025. across 15 datasets spanning five domains (mathematics, code, logic, knowledge, and affective tasks). Notably, the Avengers outperforms GPT-4.1 on nine of these datasets; it outperforms GPT-4.1 by 37.04\% on AIME, by 31.86\% on MBPP, and by 15.41\% on KORBench. When isolating the effect of the routing mechanism, the Avengers ’ clustering-based router still outperforms state-of-the-art (SOTA) performance-oriented routers (RouterDC [ 8 ] , embedLLM [ 9 ] , and MODEL-SAT [ 10 ] ), even though all those routers require neural network training (with some even involving fine-tuning LMs) and must be retrained to accommodate new models or tasks. On out-of-distribution tasks (evaluated on five additional benchmarks), the superior performance of the Avengers ’ clustering-based