The Single-Multi Evolution Loop for Self-Improving Model Collaboration Systems Shangbin Feng, Kishan Panaganti, Yulia Tsvetkov, Wenhao Yu Abstract Model collaboration — systems where multiple language models (LMs) collaborate — combines the strengths of diverse models with cost in loading multiple LMs. We improve efficiency while preserving the strengths of collaboration by distilling collaborative patterns into a single model , where the model is trained on the outputs of the model collaboration system. At inference time, only the distilled model is employed: it imitates the collaboration while only incurring the cost of a single model. Furthermore, we propose the single-multi evolution loop : multiple LMs collaborate, each distills from the collaborative outputs, and these post-distillation improved LMs collaborate again, forming a collective evolution ecosystem where models evolve and self-improve by interacting with an environment of other models. Extensive experiments with 7 collaboration strategies and 15 tasks (QA, reasoning, factuality, etc.) demonstrate that: 1) individual models improve by 8.0% on average, absorbing the strengths of collaboration while reducing the cost to a single model; 2) the collaboration also benefits from the stronger and more synergistic LMs after distillation, improving over initial systems without evolution by 14.9% on average. Analysis reveals that the single-multi evolution loop outperforms various existing evolutionary AI methods, is compatible with diverse model/collaboration/distillation settings, and helps solve problems where the initial model/system struggles to. 2 2 2 Code and data are available at https://github.com/BunsenFeng/moco_distill 1 Introduction Advancing beyond a single monolithic language model (LM), an increasing line of research focuses on model collaboration ( 13 ) , where multiple LMs collaborate and complement each other: multiple LMs “talk” and debate with each other to divide and conquer complex problems ( 11 ; 30 ) , multiple LMs are aggregated through routing systems so user queries are routed to the most fitting model ( 47 ; 16 ) , multiple LMs exchange information in the logits ( 40 ; 52 ) or model parameter ( 28 ; 15 ) space for collective generation. These methods and algorithms combine and leverage the strengths of multiple LMs, with a caveat of increased cost: the total parameter size is (often) larger, as multiple models need to be loaded at the same time; the active parameter size is also (often) larger, especially for inference-time approaches where multiple models are all used for generation. How to reduce the cost of these modular, compositional methods and systems remains an open research question. We propose to improve the efficiency of multi-LM systems while preserving the benefits of collaboration through distillation : multiple LMs first collaborate to generate outputs, and we then train a single model on the output texts/logits of the collaborative system. The distillation aims to encourage the single model to learn from and imitate the collaboration trace of the multi-LM system: at inference time, only the distilled model is employed for generation, improving the efficiency by cutting the cost from multiple LMs back to one while maintaining the benefits of collaboration. Figure 1 : We propose the single-multi evolution loop: In the multi-step , multiple language models collaborate via model collaboration algorithm \mathcal{C} to generate better responses; In the single-step , we employ knowledge distillation, where each individual LM is the student and the model collaboration system is the teacher. By alternating and iteratively executing the multi-step and single-step, multiple LLMs collaboratively evolve for better models and better model collaboration systems. Furthermore, we propose the single-multi evolution loop where multiple LMs evolve through iteratively executing this collaboration-distillation duo: models first collaborate , each of the LMs then independently distill from the collaborative outputs, and these post-distillation, enhanced, and more synergistic LMs then collaborate again, iteratively. In this way, language models evolve by interacting and learning from a system of other models : this would benefit both the capacity of the individual LMs and the collaboration system built on top of these evolving language models. We conduct extensive experiments with 3 model pool settings, 7 collaboration strategies, 3 distillation methods, and evaluate on 15 tasks spanning QA, reasoning, factuality, safety, instruction following, and more. Extensive experiments demonstrate that: 1) distilling from model collaboration improves individual LMs by 8.0% on average, and by using them in place of the multi-LM systems we improve efficiency while preserving collaborative gains; 2) the multi-LM collaboration systems also benefit from having the stronger and evolved LMs as components, improving over initial unevolved systems by 14.9% across tasks. Further analysis reveals that the single-multi evolution loop outperforms various existing evolutionary strategies by 7.7% on average, helps solve an average of 66.5% of problems in QA and reasoning where the initial models/systems struggle, and works with diverse models, collaboration methods, and distillation algorithms. Together, these efforts put forward a new paradigm of evolutionary AI systems: models self-improve and evolve by collaborating and interacting with an environment of other models. 2 Methodology 2.1 Overview Algorithm 1 Single–Multi Evolution Loop 0: Initial model pool \mathcal{M}^{(0)}=\{\mathbf{m}_{1}^{(0)},\cdots,\mathbf{m}_{n}^{(0)}\} ; Collaboration strategy \mathcal{C} (e.g., Multi-agent Debate); Distillation method \mathcal{D} (e.g., Supervised KD); Instruction dataset \mathcal{X} ; Number of iterations k 0: Evolved single model pool \mathcal{M}^{(k)} and model collaboration system \mathcal{C}(\mathcal{M}^{(k)}) 1: for t=0 to k-1 do 2: Multi-step: Collaboration 3: Initialize empty dataset \mathcal{D}^{(t)} 4: for all \mathbf{x}\in\mathcal{X} do 5: \mathbf{y}\leftarrow\mathcal{C}(\mathbf{x}\mid\mathcal{M}^{(t)}) 6: \mathcal{D}^{(t)}\leftarrow\mathcal{D}^{(t)}\cup\{(\mathbf{x},\mathbf{y})\} 7: end for 8: Single-step: Distillation 9: for all \mathbf{m}_{i}^{(t)}\in\mathcal{M}^{(t)} do 10: \mathbf{m}_{i}^{(t+1)}\leftarrow\mathcal{D}\big(\mathbf{m}_{i}^{(t)}\mid\mathcal{D}^{(t)}\big) 11: end for 12: \mathcal{M}^{(t+1)}\leftarrow\{\mathbf{m}_{1}^{(t+1)},\cdots,\mathbf{m}_{n}^{(t+1)}\} 13: end forReturn \mathcal{M}^{(k)},\mathcal{C}(\mathcal{M}^{(k)}) Given language models \mathcal{M}=\{\mathbf{m}_{1},\cdots,\mathbf{m}_{n}\} and natural language instructions \mathbf{x}\in\mathcal{X} , we propose to let models \mathcal{M} collaborate (the multi-step ), distill the collaboration outcome into individual models \mathbf{m}\in\mathcal{M} for efficiency (the single-step ), and employ the post-distillation improved LMs for collaboration again. By alternating between the multi-step and the single-step, language models collectively evolve by interacting with an environment of other models. In the multi-step , given instruction \mathbf{x}\in\mathcal{X} , we employ model collaboration strategy \mathcal{C} (e.g., multiagent debate ( 11 ) or routing ( 47 ) , Section 2.2 ) to generate a response \mathcal{C}(\mathbf{x}\mid\mathcal{M}^{(t)}) , where (t) denotes the iteration time stamp. Marginalizing over \mathbf{x}\in\mathcal{X} , we obtain a supervised fine-tuning dataset with outputs generated by the model collaboration system: \displaystyle\mathcal{D}^{(t)}=\{(\mathbf{x},\mathcal{C}(\mathbf{x}\mid\mathcal{M}^{(t)}))\}_{\mathbf{x}\in\mathcal{X}} In the single-step , we distill the collaborative outputs into each individual model (e.g., supervised distillation ( 32 ) or on-policy distillation ( 38 ; 1 ) ), thus learn from the collaboration patterns while only incurring the cost of a single model when used for inference. Specifically: \displaystyle\mathbf{m}_{i}^{(t+1)}=\mathrm{distill}(\mathbf{m}_{i}^{(t)}\mid\mathcal{D}^{(t)}) This yields a pool of improved language models \mathcal{M}^{(t+1)}=\{\mathbf{m}_{1}^{(t+1)},\cdots,\mathbf{m}_{n}^{(t+1)}\} after interacting and learning with the collaborative system of multiple models. The single-multi evolution loop is then an iterative process of alternating between the single-step and the multi-step: LMs \displaystyle\ \mathcal{M}^{(0)}\xrightarrow{\text{multi}}\text{system}\ \mathcal{C}(\mathcal{M}^{(0)})\xrightarrow{\text{single}}\text{LMs}\ \mathcal{M}^{(1)} \displaystyle\xrightarrow{\text{multi}}\cdots\xrightarrow{\text{single}}\text{LMs}\ \mathcal{M}^{(k)}\xrightarrow{\text{multi}}\text{system}\ \mathcal{C}(\mathcal{M}^{(k)}) where k denotes the number of iterations. In this way, models in \mathcal{M} collectively evolve and self-improve by interacting and collaborating with other language models. The objective of the single-multi evolution loop is twofold: \displaystyle\mathcal{M}^{(k)}\succ\mathcal{M}^{(0)}\ \ \&\ \ \mathcal{C}(\mathcal{M}^{(k)})\succ\mathcal{C}(\mathcal{M}^{(0)}) that both the individual models and the collaboration system consisting of these models should improve compared to the initial models and system, where \succ indicates better performance. We provide an overview of the single-multi evolution loop in Figure 1 and Algorithm 1 . 2.2 Multi-step: collaboration strategies \mathcal{C} In the multi-step, multiple LMs \mathcal{M} should collaborate and complement each other to generate a better response for instruction \mathbf{x}\in\mathcal{X} . We seek to cover the wide range of literature on model collaboration ( 13 ) by employing one model collaboration method per collaboration level, while experimenting with more in Section 5 . API-level collaboration features the routing and selection of different LMs for different user queries ( 47 ; 18 ; 16 ; 69 ) . At training time, we evaluate which candidate model would generate the best response to \mathbf{x} : j=\arg\max_{i=1}^{n}\mathrm{score}(\mathbf{m}_{i}(\mathbf{x})) , then train a router f on f(\mathbf{x}\mid\mathcal{M})\rightarrow\mathbf{m}_{j} . At inference-time, we employ the model f(\mathbf{x}^{\prime}\mid\mathcal{M}) to generate responses for \mathbf{x}^{\prime} . We instantiate f as a causal language model ( 47 ) . Text-level collaboration features the discussion and debate among multiple LMs to give feedback and refine responses ( 22 ; 65 ; 9 ) . We specifically employ multi-agent debate ( 11 ) where models refine responses based on others’ responses with the following prompt: Prompt 1: collaboration with multi-agent debate You are part of a team of AI assistants collaborating to answer the user’s question. Each assistant provides their own answer: use their answers to refine and improve your own answer. Question: (question) Your previous answer: (previous answer) Other assistants’ answers: (list of answers, one per row) Please provide a refined answer to the question. We run this for a few iterations and employ the best model on a held-out set to summarize all models’ responses at the last iteration for a final response. Logit-level collaboration features arithmetic with the next-token probability distributions across multiple LMs for collaborative decoding ( 40 ; 52 ) . We specifically employ logit fusion: for model \mathbf{m}_{i} ’s next-token distribution \mathbf{p}_{i} , we aggregate across models \mathbf{p}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{p}_{i} and then decode text from \mathbf{p} . This requires models in \mathcal{M} to share mostly similar vocabulary/tokenization (e.g., LoRA adapters ( 26 ) finetuned from a common base model). Weight-level collaboration features the merging and searching in the model parameter space across multiple LMs ( 28 ; 59 ; 15 ) . We specifically employ dare-ties model merging ( 60 ; 62 ) due to their strong numerical properties. \mathcal{C}(\mathbf{x}\mid\mathcal{M}) in this case is then the response generated by the merged model. This requires models in \mathcal{M} to share mostly the same architecture (e.g., LoRA adapters finetuned from a common base model). With the four representative model collaboration strategies as \mathcal{C} , we aim to investigate whether the single-model evolution loop is a broadly applicable idea and whether certain modes of collaboration would be especially helpful. 2.3 Single-step: distillation strategies In the single-step, we independently employ each model in \mathcal{M} as student and the collaboration system \mathcal{C}(\mathbf{x}\mid\mathcal{M}) as teacher for knowledge distillation (KD), so models could interact with other models and learn from the collaboration outcome to collectively evolve and self-improve. We consider the following distillation strategies: Supervised KD : model \mathbf{m} is supervised fine-tuned on the generated texts of the collaboration system \{(\mathbf{x},\mathcal{C}(\mathbf{x}\mid\mathcal{M}^{(t)}))\}_{\mathbf{x}\in\mathcal{X}} . This is most flexible and efficient, without making assumptions about tokenization. Multi-student KD ( 35 ) : two findings motivate an adjusted version of supervised KD: 1) the skill gap between \mathbf{m} and \mathcal{C}(\mathbf{x}\mid\mathcal{M}) is sometimes large for effective distillation, especially in early iterations of the single-multi evolution loop; 2) since all models employ the same teacher/data for distillation, the variation/diversity across models might be reduced. As such, we formulate multi-student KD, where the models are supervised fine-tuned on an \alpha:\beta:\gamma mixture of collaboration outputs, outputs of the strongest student (with a smaller student-student capability gap than teacher-student), and outputs of itself (to preserve its specialization), respectively \{(\mathbf{x},\mathcal{C}(\mathbf{x}\mid\mathcal{M}^{(t)}))\}_{\mathbf{x}\in\mathcal{X}} , \{(\mathbf{x},\mathbf{m}_{\textit{best}}(\mathbf{x}))\}_{\mathbf{x}\in\mathcal{X}} , and \{(\mathbf{x},\mathbf{m}(\mathbf{x})\}_{\mathbf{x}\in\mathcal{X}} . Logit-based KD : we additionally employ logit-based distillation (minimizing the KL divergence between student/teacher logit distributions, in addition to cross-entropy loss of token prediction), specifically on-policy distillation ( 1 ; 48 ) (token from student but logit from teacher). These methods are bounded by tokenization assumptions between the teacher and the student for logit calculations and may not be applicable to every collaboration strategy \mathcal{C} as the teacher, potentially extensible by reconciling tokenization differences. Table 1 : Performance of the single-multi evolution loop with different model pools, collaboration methods, and distillation settings. We highlight improvements in orange ( \mathrm{single}_{t=n}>\mathrm{single}_{t=0} , \mathrm{multi}_{t=n}>\mathrm{multi}_{t=0} ). Best in bold and second-best in underline . Results show that the single-multi evolution loop is a broadly effective idea, compatible with diverse models and collaboration strategies, bringing performance gains to both individual models and collaboration systems in 91.9% settings by 8.0% and 14.9% on average. pool 1: specialized LLMs general QA reasoning knowledge safety science inst. agieval arc mmlu-r bbh gsm8k math gpqa-dia tablemwp theoremqa wikidyk popqa blend truthful sciriff hinterest single, t=0 51.10 (0.72) 81.66 (3.94) 65.57 (2.21) 38.53 (7.13) 82.40 (2.36) 77.41 (0.79) 31.99 (1.54) 61.87 (2.01) 27.92 (1.91) 4.43 (0.35) 15.79 (1.28) 72.23 (1.47) 56.83 (5.49) 53.48 (6.20) 1.80 (0.68) multi, t=0 api: trained router 51.04 81.06 65.10 39.30 79.64 74.93 27.27 63.20 34.00 4.87 16.27 70.70 54.46 52.52 2.24 text: multiagent debate 52.77 68.94 55.50 45.90 76.50 81.28 27.27 42.50 30.25 3.33 14.66 67.10 45.06 54.68 2.98 logit: logit fusion 43.43 83.79 73.70 13.10 55.70 68.05 34.34 43.20 21.50 5.97 7.43 61.60 51.70 47.48 -3.14 weight: dare ties 52.16 82.17 69.90 40.90 83.90 71.40 29.29 60.80 25.00 5.08 17.77 73.00 51.28 64.75 0.03 supervised-KD api single, t=n 62.31 (0.40) 86.58 (0.63) 71.10 (0.78) 40.40 (0.36) 90.37 (0.76) 81.76 (0.22) 30.98 (1.17) 72.27 (1.67) 33.17 (1.63) 5.60 (0.24) 18.14 (0.41) 78.83 (0.12) 67.37 (0.50) 56.59 (1.10) 15.27 (0.50) multi, t=n 62.28 86.52 70.60 40.90 89.90 82.22 33.33 73.60 36.75 5.27 20.92 79.00 68.88 59.71 15.14 text single, t=n 55.42 (0.53) 84.78 (0.69) 68.20 (0.70) 42.20 (0.26) 90.70 (1.00) 81.97 (0.63) 30.30 (1.01) 72.53 (0.46) 33.58 (1.70) 5.60 (0.22) 18.10 (0.34) 78.97 (0.61) 67.04 (0.80) 57.31 (0.42) 15.28 (0.38) multi, t=n 57.96 72.27 60.50 51.60 85.00 82.11 25.25 69.60 39.06 5.92 16.70 74.80 68.07 58.27 9.42 logit single, t=n 60.64 (1.36) 85.61 (0.56) 68.40 (0.40) 40.60 (0.66) 68.13 (3.71) 71.61 (7.13) 38.62 (3.25) 71.20 (1.22) 24.58 (3.17) 5.30 (0.14) 17.88 (0.21) 79.07 (0.35) 67.86 (1.04) 56.83 (0.72) -0.43 (0.21) multi, t=n 35.99 86.18 27.10 43.19 56.50 71.73 40.94 48.80 21.50 8.10 18.55 76.20 66.29 58.27 0.74 weight single, t=n 44.87 (0.95) 85.64 (0.39) 69.63 (0.57) 41.13 (1.36) 90.70 (0.72) 81.90 (0.55) 32.66 (2.10) 71.73 (1.67) 33.00 (1.09) 5.25 (0.22) 18.23 (0.13) 78.80 (0.56) 67.21 (0.47) 57.07 (1.66) 15.15 (0.68) multi, t=n 57.18 83.19 73.90 56.14 82.90 77.30 25.25 78.77 27.75 7.61 19.78 74.85 69.60 70.60 12.24 multi-KD api single, t=n 61.19 (1.10) 86.29 (0.20) 70.8 (0.82) 40.23 (0.49) 89.77 (0.49) 81.97 (0.63) 34.01 (1.17) 70.67 (0.46) 33.92 (1.26) 5.36 (0.19) 16.08 (2.74) 78.70 (0.26) 66.56 (0.57) 56.35 (0.42) 14.85 (0.44) multi, t=n 66.77 87.03 71.30 58.20 91.20 77.09 32.32 72.80 34.75 5.79 15.82 79.60 68.23 57.55 15.53 text single, t=n 59.17 (0.79) 86.18 (0.96) 69.03 (0.81) 40.83 (1.43) 90.77 (0.21) 82.15 (0.59) 34.68 (4.77) 73.07 (1.22) 34.42 (1.63) 5.80 (0.24) 18.30 (0.06) 79.10 (0.50) 67.59 (0.58) 57.55 (0.72) 15.05 (0.40) multi, t=n 55.19 73.81 79.20 50.10 85.20 83.37 31.31 71.20 41.40 6.01 16.14 71.60 66.61 58.27 10.01 logit single, t=n 53.84 (1.48) 87.34 (0.27) 69.97 (0.25) 40.50 (0.44) 67.07 (2.65) 61.96 (7.56) 26.26 (6.14) 71.20 (1.60) 26.00 (0.90) 5.64 (0.23) 18.70 (0.25) 78.47 (0.45) 67.21 (1.04) 56.59 (1.50) 4.47 (1.69) multi, t=n 43.08 84.73 45.24 17.10 60.50 46.23 37.37 53.60 21.50 1.20 6.04 77.00 65.96 57.55 8.65 weight single, t=n 60.73 (0.17) 86.63 (0.50) 71.67 (0.55) 39.93 (0.68) 89.67 (0.81) 82.04 (0.79) 29.63 (4.98) 72.53 (1.67) 34.33 (0.88) 5.62 (0.30) 18.82 (0.45) 78.40 (0.30) 67.31 (0.61) 56.35 (0.42) 15.31 (0.29) multi, t=n 56.57 87.46 71.30 40.40 91.20 79.29 34.34 70.40 33.50 5.08 17.67 78.80 68.40 57.55 18.73 pool 2: general-purpose LLMs single, t=0 38.87 (6.28) 61.77 (7.69) 50.87 (8.47) 32.40 (9.72) 83.47 (6.67) 72.97 (10.21) 25.93 (3.55) 60.80 (10.82) 27.08 (5.20) 4.73 (2.43) 16.47 (9.04) 57.37 (3.80) 50.08 (7.97) 48.68 (8.90) 2.09 (1.86) multi, t=0 api: trained router 38.33 62.65 52.20 54.20 72.22 74.54 25.25 49.60 30.25 7.91 17.41 (9.38) 65.70 49.81 35.70 0.26 text: multiagent debate 54.67 63.65 51.60 60.40 83.30 74.58 22.22 68.00 30.00 5.53 19.78 68.00 57.37 55.40 6.74 supervised-KD api single, t=n 40.66 (5.84) 64.56 (9.88) 53.57 (4.41) 35.50 (5.30) 83.57 (6.33) 77.01 (9.66) 30.30 (2.67) 62.40 (11.00) 29.08 (6.42) 7.08 (2.62) 19.96 (9.66) 59.20 (3.98) 52.78 (8.43) 53.00 (5.45) 2.80 (1.88) multi, t=n 41.75 70.65 55.20 67.10 79.80 78.49 37.23 67.60 39.90 12.75 21.33 71.40 57.00 68.60 1.37 text single, t=n 40.54 (3.83) 64.48 (4.59) 52.50 (3.26) 34.63 (9.56) 83.40 (6.08) 76.43 (10.00) 27.95 (4.55) 63.47 (10.41) 28.58 (4.58) 5.27 (3.11) 19.08 (9.64) 58.80 (3.74) 51.32 (8.02) 53.24 (8.72) 2.71 (1.88) multi, t=n 57.53 65.10 53.22 61.70 85.80 77.30 34.34 72.00 32.75 5.65 20.06 71.70 59.16 62.59 7.05 multi-KD api single, t=n 45.70 (6.86) 65.48 (7.49) 53.30 (8.74) 34.30 (6.45) 83.67 (6.05) 76.03 (9.66) 28.96 (2.33) 62.93 (10.13) 30.58 (3.01) 5.22 (2.50) 19.58 (8.92) 58.77 (3.66) 51.38 (7.31) 52.52 (8.11) 2.66 (1.91) multi, t=n 38.49 66.20 53.60 65.30 78.30 75.59 27.07 72.40 31.32 8.55 22.15 71.20 57.74 65.90 0.53 text single, t=n 45.04 (6.36) 65.10 (5.51) 53.10 (7.27) 35.90 (3.28) 84.50 (5.21) 75.84 (10.09) 29.63 (0.58) 61.33 (10.07) 29.17 (4.13) 5.23 (2.37) 19.20 (9.55) 58.97 (3.95) 51.22 (7.49) 53.24 (6.86) 2.50 (1.88) multi, t=n 57.27 67.06 52.50 65.30 86.30 80.13 36.36 72.00 32.75 6.17 20.35 72.50 61.10 62.59 7.23 3 Experiment Settings Models and Implementation We employ two model pools for collaboration and distillation: three specialized language models based on Qwen2.5-7B-Instruct , separately finetuned ( 30 ) on three domains of SFT data in Tulu-v3 ( 34 ) ( pool #1 ), and three general-purpose language models ( pool #2 : Qwen/Qwen2.5-7B-Instruct , meta-llama/Llama-3.1-8B-Instruct , deepseek-ai/DeepSeek-R1-Distill-Qwen-7B ). We employ all four collaboration strategies in Section 2.2 (trained router, multiagent debate, logit fusion, and dare-ties) for model pool #1, as they share the same tokenization and architecture, and model pool #2 is only compatible with the first two model collaboration approaches. In trained router, we instantiate the router f as Qwen2.5-7B-Instruct and train it with 1e-5 learning rate for 5 epochs; In multi-agent debate, we run the debate and run process three times for collaboration; In dare-ties merging, we employ the implementation of MergeKit ( 20 ) and assign equal weight to models. We by default employ 512 max new tokens, temperature \tau=0.7 , and top-p p=0.9 sampling for text generation. We use the implementation of MoCo ( 12 ) for model collaboration algorithms. We by default employ both supervised and multi-student knowledge distillation in Section 2.3 , while we extend to on-policy distillation in Section 5 . We employ \alpha:\beta:\gamma=1:1:1 for multi-student KD. We by default execute the single-multi evolution loop for k=3 iterations and retain the best model/collaboration system based on the dev set. Data and Evaluation We employ 15 datasets spanning six domains for evaluation: • QA: AGIEval ( 70 ) , ARC-challenge ( 7 ) , MMLU-redux ( 19 ) • reasoning: BigBench-hard ( 55 ) , GSM8k ( 8 ) , MATH ( 24 ) , TheoremQA ( 5 ) , GPQA-diamond ( 50 ) , TableMWP ( 42 ) ) • knowledge: WikiDYK ( 64 ) , PopQA ( 43 ) , BLEND ( 46 ) • misc: safety (CocoNot ( 2 ) ), science (Sciriff ( 57 ) ), and instruction following (human interest ( 15 ) ) We by default sample 1k data points for both dev and test sets for large datasets. We report both the \mathrm{avg}\pm\mathrm{std} for the single models and the performance of the multi-model collaboration system. 4 Results We present the performance of the single-multi evolution loop with various model pools, collaboration strategies, and distillation settings in Table 1 . The single-multi evolution loop is effective. Averaging across datasets, collaboration strategies, and distillation settings, the single-multi evolution loops improve both individual models and the collaboration system they form by 8.0% and 14.9% on average. This indicates that models collectively evolve by interacting with an environment of other models and learning from the collaborative outcome. Collaboration strategies such as routing and merging are more compatible. In setting 1, API-level routing (53.39), weight-level model merging (52.72), text-level multiagent debate (50.29), and API-level logit fusion (42.86) achieved decreasing performance. We posit that the best aggregate of responses or model weights serves as a stronger teacher. This also indicates that the single-multi evolution loops’s success depends on model collaboration strategies: we experiment with more collaboration strategies in Section 5 . The single-multi evolution loop benefits from better distillation and model diversity. Multi-student knowledge distillation on average outperforms simple supervised distillation by 4.67% on average, indicating that the single-multi evolution loop benefits from stronger distillation methods and we further extend to on-policy distillation in Section 5 . In addition, the performance gains of API- and text-level approaches in model pool 1 (specialized LMs, 19.62%) are higher than model pool 2 (general-purpose LMs, 10.03%): while both decent, this indicates the single-multi evolution loop benefits from the complementary strength of and specialized language models. More effective on reasoning and knowledge. On the three domains with more than one dataset, the single-multi evolution loops achieve 16.84% (reasoning), 12.67% (knowledge), and 7.52% (QA) gains. This indicates that by empowering models to interact and collaborate with an environment of other models and learn from the collaborative outcome, models benefit from the complementary reasoning patterns and knowledge capacity of other models. Figure 2 : Comparing the single-multi evolution loop with existing evolution strategies. Our strategy consistently outperforms the three methods by 7.7% on average. 5 Analysis Comparison with Other Evolution Methods We compare the single-multi evolution loop with three existing methods of iterative evolution: multi-agent debate ( 11 ) , where model responses are refined based on other models’ responses; multi-agent fine-tuning ( 54 ) , where models are trained collectively via critique and debate; sparta alignment ( 30 ) , where models evaluate and align each other under game theory settings. Figure 2 shows the performance under different evolution settings with model pool #1 and three datasets: the single-multi evolution loop outperforms the three existing methods by 7.7% on average, indicating that the strategy of introducing deeper collaboration and employing a collaborative system as the teacher for distillation yields better collective evolution of diverse language models. On-Policy Distillation Section 4 show that multi-student distillation generally outperforms simple supervised distillation, indicating that the single-multi evolution loop benefits from employing better distillation methods. We further extend this to on-policy distillation ( 1 ) , specifically GOLD ( 48 ) , and employ dare-ties model merging the collaboration strategy \mathcal{C} and model pool #1. 3 3 3 This ensures that the teacher is a merged single model, compatible with the on-policy setting requiring logit information. Results in Table 2 show that on-policy distillation is even better than multi-student and supervised knowledge distillation, outperforming them by up to 5.2%. We envision further exploration on the deep literature of knowledge distillation research, how that might benefit model collaboration and model evolution, and how to flexibly enable collaborative systems of multiple models as the teacher in distillation methods. Table 2 : Employing on-policy distillation in the single-multi evolution loop yields further gains compared to multi-student and simple supervised distillation, indicating that the broad literature of knowledge distillation methods could be valuable for model collaboration and evolution. MMLU-redux MATH TheoremQA BLEND Single Multi Single Multi Single Multi Single Multi Initial 65.57 69.90 77.41 71.40 27.92 25.00 72.23 73.00 Supervised KD 69.63 73.90 81.90 77.30 33.00 27.75 78.80 74.85 Multi-Student KD 71.67 71.30 82.04 79.29 34.33 33.50 78.40 78.80 On-Policy KD 73.17 74.20 83.30 79.92 34.67 35.25 79.62 80.45 Figure 3 : How does the skills dynamic change through evolution iterations, i.e. how many problems could be solved by at least one model individually or the multi-LLM collaboration system. Results indicate better and more synergistic models and systems. Table 3 : Results with three additional model collaboration strategies show that the single-multi evolution loop is a broadly effective idea compatible with diverse model collaboration algorithms. TableMWP SciRiff single, t=0 58.93 (10.56) 51.80 (8.84) LLM Blender multi, t=0 78.40 56.83 single, t=n 61.87 (10.00) 54.44 (6.12) multi, t=n 80.00 58.99 Multiagent Finetuning multi, t=0 76.00 51.80 single, t=n 60.80 (9.23) 53.04 (7.76) multi, t=n 79.20 56.12 Graph Routing multi, t=0 64.80 52.52 single, t=n 61.60 (12.87) 53.80 (8.11) multi, t=n 71.20 58.27 Table 4 : The single-multi evolution loop brings consistent improvements to the three official variants of Qwen-2.5-7B . MMLU-redux MATH TheoremQA BLEND single, t=0 54.70 (4.99) 75.77 (5.76) 29.08 (3.64) 58.33 (7.14) multi, t=0 48.70 77.20 31.00 67.00 single, t=n 56.90 (5.28) 76.85 (4.83) 32.00 (2.78) 61.63 (6.49) multi, t=n 52.50 80.86 33.50 69.30 Skills Change of Single/Multi Given an evaluation problem, whether the single models or the multi-LLM system can correctly solve the problem varies. We use “single yes/no” to denote whether any of the single models can individually solve the problem, and use “multi yes/no” to denote whether the model collaboration can do so. This creates a 2\times 2 confusion matrix highlighting the skill dynamics between individual models and the model collaboration system. We visualize the change of the skill dynamics across single-multi evolution iterations in Figure 3 . 1) There are consistently fewer “single no, multi no” cases, indicating that the single-multi evolution loop helps models solve problems that were impossible to them initially; 2) There are consistently more “single no, multi yes” cases, indicating that the ceiling of collaboration is also raised due to more synergistic models through the collective evolution process. Table 5 : The average and best performance of individual models, before and after evolution. This indicates that the single-multi evolution loop improves models both on average and on the best model’s utility. MATH GPQA-dia TheoremQA single, t=0, avg 72.97 (10.21) 25.93 (3.55) 27.08 (5.20) single, t=0, best 81.69 30.30 32.75 single, t=n, avg 75.84 (10.09) 29.63 (1.58) 29.17 (4.13) single, t=n, best 84.26 32.32 35.00 Figure 4 : The performance of single individual models and multi-model systems across iterations, with a pool of three different sizes of Qwen-2.5 . We employ multi-student KD and the BLEND dataset for evaluation. More Collaboration Strategies \mathcal{C} In Section 4 , we employ four model collaboration strategies to instantiate the single-multi evolution loop: trained router, multi-agent debate, logit fusion, and dare-ties model merging. We additionally test out the single-multi evolution loop with three more advanced model collaboration algorithms: LLM blender ( 29 ) , where the responses of multiple LLMs and reranked and merged by training separate LMs; multi-agent fine-tuning ( 54 ) , a training-based enhancement of multi-agent debate; and graph router ( 16 ) , employing graph neural networks as the routing strategy. Table 3 shows that the single-multi evolution loop continues to bring improvements to both individual LMs and model collaboration systems with these three collaboration strategies, indicating its broad compatibility and effectiveness with diverse model collaboration protocols. Table 6 : Qualitative examples of model-generated responses to complex problems in the TheoremQA dataset. Model Version Generated Text Question Consider x(t) to be given as, x(t)=\cos(1000\pi t) . Let the sampling frequency be 2000\mathrm{~Hz} . Does aliasing occur? Single, t=0 Yes, aliasing will occur since the frequency of the signal is 1000 Hz which is equal to the Nyquist frequency, and the sampling frequency is also 2000 Hz which is twice the Nyquist frequency. Therefore, the signal cannot be accurately reconstructed from its samples due to aliasing. ✗ Single, t=n Step 1: Determine the Frequency of the Signal The given signal is: x(t)=\cos(1000\pi t) . The general form of a cosine function is: \cos(2\pi ft) . By comparing this with the given signal, we can see that: 2\pi f=1000\pi , which results in f=500\text{ Hz} . So, the frequency of the signal x(t) is 500\text{ Hz} . Step 2: Determine the Nyquist Frequency The Nyquist frequency is half of the sampling frequency. Given the sampling frequency is 2000\text{ Hz} , the Nyquist frequency is: f_{\text{Nyquist}}=\frac{2000\text{ Hz}}{2}=1000\text{ Hz} . Step 3: Compare the Signal Frequency with the Nyquist Frequency For a signal to avoid aliasing, its frequency must be less than or equal to the Nyquist frequency. In this case: The signal frequency f=500\text{ Hz} and the Nyquist frequency f_{\text{Nyquist}}=1000\text{ Hz} . Since 500\text{ Hz}\leq 1000\text{ Hz} , the signal’s frequency is within the non-aliasing range. Conclusion: No aliasing occurs because the signal frequency 500\text{ Hz} is below the Nyquist frequency 1000\text{ Hz} . ✓ Question A cascade of n identical independent binary symmetric channels each with raw error probability p , and 0