Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation
Zhenglun Kong, Zheng Zhan, Shiyue Hou, Yifan Gong, Xin Meng, Pengwei Sui, Peiyan Dong, Xuan Shen, Zifeng Wang, Pu Zhao, Hao Tang, Stratis Ioannidis, Yanzhi Wang · 2025
Read Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation by Zhenglun Kong, Zheng Zhan, Shiyue Hou, Yifan Gong, Xin Meng, Pengwei Sui, Peiyan Dong, Xuan Shen, Zifeng Wang, Pu Zhao, Hao Tang, Stratis Ioannidis, Yanzhi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23844.txt.
Introduction
Combining the skills of several large language models into one model is hard, because ensembles take huge memory and merging weights is rigid. Zhenglun Kong and co-authors proposed a framework for that task in 2025. A selection network picks the most relevant source models for each need, which cuts clashes between their knowledge.
A weighted fusion step then blends the chosen knowledge by each source model's strengths. A feedback-driven loss keeps the selector from always picking the same few sources. In tests, the method cut knowledge interference by up to half compared with earlier approaches. The authors present it as a stable, scalable way to keep improving a single model.
Catalog record
| CREATOR | Zhenglun Kong, Zheng Zhan, Shiyue Hou, Yifan Gong, Xin Meng, Pengwei Sui, Peiyan Dong, Xuan Shen, Zifeng Wang, Pu Zhao, Hao Tang, Stratis Ioannidis, Yanzhi Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23844
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation Zhenglun Kong, Zheng Zhan, Shiyue Hou, Yifan Gong, Xin Meng, Pengwei Sui, Peiyan Dong, Xuan Shen, Zifeng Wang, Pu Zhao, Hao Tang, Stratis Ioannidis, Yanzhi Wang Abstract Large language models (LLMs) have shown remarkable promise but remain challenging to continually improve through traditional finetuning, particularly when integrating capabilities from other specialized LLMs. Popular methods like ensemble and weight merging require substantial memory and struggle to adapt to changing data environments. Recent efforts have transferred knowledge from multiple LLMs into a single target model; however, they suffer from interference and degraded performance among tasks, largely due to limited flexibility in candidate selection and training pipelines. To address these issues, we propose a framework that adaptively selects and aggregates knowledge from diverse LLMs to build a single, stronger model, avoiding the high memory overhead of ensemble and inflexible weight merging. Specifically, we design an adaptive selection network that identifies the most relevant source LLMs based on their scores, thereby reducing knowledge interference. We further propose a dynamic weighted fusion strategy that accounts for the inherent strengths of candidate LLMs, along with a feedback-driven loss function that prevents the selector from converging on a single subset of sources. Experimental results demonstrate that our method can enable a more stable and scalable knowledge aggregation process while reducing knowledge interference by up to 50% compared to existing approaches. 1 1 1 Code is avaliable at Github . 1 Introduction The emergence of large language models (LLMs) has revolutionized various domains, catalyzing the development of many specialized models zhang2023alpacareinstructiontuned ; christophe2024med42 ; koala_blogpost_2023 ; leng2023chinese-vicuna ; li2023starcoder ; li2025comprehensive ; wang2025systematic ; li2024distinct . Training a new LLM from scratch is prohibitively expensive in terms of computational resources and data requirements. As a result, the prevalent strategy is to finetune existing models using localized or task-specific data. While traditional finetuning can incrementally enhance a model’s performance within a specific domain, it often fails to leverage the rich, domain-specific expertise embedded in other LLMs, especially when relevant datasets are inaccessible or require extensive pre-processing li2024towards . Furthermore, for organizations that have invested heavily in developing and tailoring their own LLMs, replacing the current model with a new, pre-trained one introduces challenges such as re-adaptation, retraining, and potentially significant costs to maintain alignment with their unique requirements. Therefore, we focus on building a stronger LLM by incorporating knowledge from multiple specialized models. Instead of finetuning a single model in isolation, we aggregate knowledge from various LLMs to enhance performance in a stable, efficient, and scalable way. This approach preserves the strengths of the existing model while infusing it with complementary knowledge, ensuring that the final model is both highly capable and aligned with specific needs. Figure 1: Scaling number of fusion candidates. We show accuracy (histogram, higher the better) and the percentage of tasks degrading the baseline (Line chart, lower the better) when integrating three, four, and five LLMs on the BBH and MMLU benchmarks. Dotted lines represent the baseline. Existing solutions, such as ensemble methods jiang2023llm ; lu2023routing ; zhong2025enhancing , enhance prediction performance by aggregating outputs from multiple models but require additional memory and increased inference time due to operating several models simultaneously. Another method involves merging several neural networks into a single network within the parameter space jin2022dataless . This generally requires ensuring uniform architecture and depends on manually configured weight merging or adding additional layers. Additionally, Mixture of Expert (MoE) structures such as Mistral-7bx8 jiang2024mixtral , address some inference and weight-sharing issues, but still face long inference times, homogeneous architectures, and larger model sizes. FuseLLM wan2024knowledge and FuseChat wan2024fusechat attempted to integrate the knowledge of multiple source LLMs using generated probability distribution matrices. However, these approaches suffer from interference and performance degradation in various tasks compared to the original target model due to suboptimal model selection and uncontrolled fusion processes. To overcome the limitations of existing LLM integration approaches, we propose a dynamic framework that adaptively selects LLMs for integration. Given a diverse set of source LLMs with heterogeneous structures, we introduce an adaptive selection network, a learnable mechanism that explicitly evaluates and selects the best-performing source LLMs based on their important scores, thereby alleviating interference issues typically associated with model fusion. The scores are computed based on the performance of each model across a predefined set of tasks. Our framework provides flexibility in the number of LLMs selected during this process. To improve the knowledge aggregation process, we propose a dynamic weighted fusion strategy that considers the intrinsic characteristics of candidate LLMs during fusion. The assigned weights are derived from the score evaluations, allowing the fusion process to prioritize models that are more likely to enhance the overall performance of the composite LLM. The selector often converges to a state in which it consistently assigns large weights to a small subset of candidates. To mitigate this, we introduce a feedback-driven loss function that optimizes the training of our adaptive selection network and guides the selection of candidates. Our method ensures stable and scalable integration of LLMs while maintaining both efficiency and effectiveness despite model diversity, as shown in Fig. 1 (Detailed analysis in Sec. 6 : Model Scaling Results). It achieves this without increasing the parameter size or computation of the target model, making it more efficient compared to traditional methods. Our contributions are as follows: • We find that merely increasing the number of fusion candidates and expanding the source model pool does not necessarily enhance the fusion process, a selective strategy is more effective in minimizing knowledge interference. • We propose a novel dynamic integration framework that adaptively selects LLMs for integration, leveraging an adaptive selection network, a dynamic weighted fusion strategy, and a feedback-driven loss function to alleviate interference issues and enhance performance. • Our model shows improvement in accuracy across multiple benchmarks as more models are integrated, while reducing knowledge interference by up to 50% compared to previous methods. 2 Related Work Model Integration. Research on model integration has evolved into distinct categories, each addressing different aspects of combining models wang2024learn ; wang2024rehearsal : 1) Ensemble: LLM-Blender jiang2023llm uses ensemble techniques to enhance performance by combining outputs from multiple models. Figure 2: Overall framework: Multiple LLMs are evaluated and selected based on performance by an adaptive selection network. Top candidates then proceed through a dynamic weighted fusion process guided by a feedback loss to enhance the ability of the target LLM. The lower right shows results on CommonSense, MMLU, and Big-Bench Hard benchmark. This process includes inferring all candidate models and then ranking them, which can be resource-intensive and slow. 2) Weight Merging: Zipit stoica2023zipit merges partial layers of two models without additional training, creating a multi-head model for various tasks. rame2022diverse ; arpit2022ensemble ; wortsman2022model employ weighted averaging methods. TIES-Merging yadav2024ties eliminates parameter interference among multiple models by removing delta parameters with low magnitudes and merging parameters with consistent signs. zhang2023composing compose models through linear arithmetic operations in the weight space. These techniques are typically limited to models with identical architectures. 3) Knowledge Fusion: FuseLLM wan2024knowledge and FuseChat wan2024fusechat focuses on fusing the probability distributions from various LLM candidates, integrating them into a single base LLM, blending knowledge across models.