Bench-CoE: a Framework for Collaboration of Experts from Benchmark

Yuanshuai Wang, Xingjian Zhang, Jinkun Zhao, Siwei Wen, Peilin Feng, Shuhao Liao, Lei Huang, Wenjun Wu · 2024

Read Bench-CoE: a Framework for Collaboration of Experts from Benchmark by Yuanshuai Wang, Xingjian Zhang, Jinkun Zhao, Siwei Wen, Peilin Feng, Shuhao Liao, Lei Huang, Wenjun Wu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04167.txt.

Introduction

Yuanshuai Wang and co-authors present a 2024 paper on teaming up specialist language models instead of relying on one. Many expert models now exist, each strong at different tasks, and each has tests that score its skills. Their framework uses those test scores to route each job to the best expert.

A router sends each query to a fitting expert, trained on benchmark results. The authors try routing at the level of single queries and at the level of whole subjects. Tests on language and image tasks show the team beat any single model overall. The work offers a base for research on model teams.

Catalog record

CREATORYuanshuai Wang, Xingjian Zhang, Jinkun Zhao, Siwei Wen, Peilin Feng, Shuhao Liao, Lei Huang, Wenjun Wu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04167

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bench-CoE: a Framework for Collaboration of Experts from Benchmark Yuanshuai Wang, Xingjian Zhang, Jinkun Zhao, Siwei Wen, Peilin Feng, Shuhao Liao, Lei Huang, Wenjun Wu Abstract Large Language Models (LLMs) are key technologies driving intelligent systems to handle multiple tasks. To meet the demands of various tasks, an increasing number of LLMs-driven experts with diverse capabilities have been developed, accompanied by corresponding benchmarks to evaluate their performance. This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations to achieve optimal performance across various tasks. Bench-CoE includes a set of expert models, a router for assigning tasks to corresponding experts, and a benchmark dataset for training the router. Moreover, we formulate Query-Level and Subject-Level approaches based on our framework, and analyze the merits and drawbacks of these two approaches. Finally, we conduct a series of experiments with vary data distributions on both language and multimodal tasks to validate that our proposed Bench-CoE outperforms any single model in terms of overall performance. We hope this method serves as a baseline for further research in this area. The code is available at https://github.com/ZhangXJ199/Bench-CoE . 1 Introduction Large Language Models (LLMs) are capable of performing various natural language processing (NLP) tasks, by using auto-regressive prediction conditioned by the task prompt Radford et al. 2019 ; Brown et al. 2020 . LLMs’ ability in describing and unifying tasks make them being the key components in current visual understanding tasks, which gives rise to the Large Multimodal Models (LMMs) Liu et al. 2023 ; Zhu et al. 2024 . While these models are able to perform all kinds of visual and language tasks, they may have different expertise and show significant diversity in performance for different tasks. We refer to these LLMs or LMMs models as experts in this paper. One interesting question arises that how can we effectively identify and exploit the abilities of different experts. Figure 1 : The framework of Bench-CoE. Our Bench-CoE Framework directly trains the router based on the benchmark, utilizing either subject-level or query-level labels for task assignment. This approach enables Bench-CoE to seamlessly integrate multiple expert models without incurring additional training costs, while simultaneously enhancing task performance. As the capabilities of experts gradually improve, benchmark tests have also become more complex and diverse Tjong Kim Sang 2002 ; Bowman et al. 2015 ; Rajpurkar et al. 2016 . For example, there are specific domain benchmarks such as GSM8K Cobbe et al. 2021 for assessing mathematical abilities and VCR Zhang et al. 2024 for evaluating visual reasoning. Additionally, there are benchmarks like MMLU Wang et al. 2024 for assessing language task multi-subjects reasoning abilities and MMMU Yue et al. 2024 for evaluating multi-domain reasoning in multimodal tasks, both designed to evaluate expert models’ capabilities across multiple subjects or domains. These reflect the evolution of expert model evaluation from single to diverse criteria, and they provide insights into the strengths and weaknesses of experts across different tasks. We propose the Bench-CoE framework, which enables expert collaboration (CoE) by effectively leveraging the strengths of different experts from benchmark evaluations. Bench-CoE includes a set of expert models, a router for assigning tasks to corresponding experts, and a benchmark dataset for training the router. Based on this framework, we first formulate the Query-Level Bench-CoE. The Query-Level approach is an abstraction of previous methods, such as Ong et al. 2024 ; Lu et al. 2024 . This approach requires evaluating the performance of different expert models for each query. Although it provides detailed information for training the router to select experts, it is expensive in terms of label creation and computational cost, and it struggles to generalize to tasks outside the data distribution. Is there a way to maintain generalization without additional cost? We analyzed that the key to the problem is obtaining labels for query assignment, and the performance of expert models across different subjects in benchmark tests actually serves as a type of label. Compared to the fine-grained Query-Level label, the more coarse-grained Subject-Level label is more likely to maintain the generalization of expert models. We further introduce Subject-Level Bench-CoE, which effectively exploits coarse-grained Subject-Level label from benchmark evaluations, as current benchmarks typically provide subject-level evaluation results Wang et al. 2024 ; Yue et al. 2024 . During router training, each query in the benchmark dataset is used to train the router, which encourages the router to assign the query to the expert model that performs best in the subject of the query. We evaluate the effectiveness and generalization of the query-level and subject-level routing learning mechanisms through a series of experiments. The experimental results show that both routing mechanisms improve the performance of the Bench-CoE model over using the best individual model. Specifically, the query-level router performs better on in-distribution data but is prone to overfitting on out-of-distribution data due to the fine-grained routing decisions. In contrast, the subject-level router demonstrates stronger generalization on out-of-distribution data, showcasing its better adaptability and robustness. Figure 2 : Comparison of routing methods in LLMs combination: (a) The MoE model utilizes multiple FFNs as expert modules during inference. (b) The Parallel-Inference-CoE model requires each query to pass through all experts during inference. (c) Although only the best expert is selected for inference, all expert models need to be tested during training to obtain labels. (d) Our Bench-CoE model only uses benchmark evaluation information to generate labels during router training, without extra costs, and uses only the best expert during inference. To summarize, our main contributions are as follows: • We propose Bench-CoE, a simple and efficient pipeline for combining and routing LLM-driven experts, which achieves flexible and efficient task routing without relying on extensive labeled data and large-scale training. • We utilize the performance of each model on benchmarks to select the LLMs to be combined and construct subject-level and query-level datasets to support accurate routing for various specialized tasks. • Experiments demonstrate that the proposed method outperforms single models in multi-task scenarios, enhancing cross-domain multi-task processing performance with almost negligible inference cost. 2 Related Work Recent research has focused on the efficient utilization and combination of LLMs in multi-task environments, aiming to meet diverse query requirements and manage resource constraints effectively. Researchers have investigated several routing and integration strategies, developing methods that improve the task-handling efficiency of LLMs. These existing methods can be broadly categorized as follows: 2.1 Mixture of Experts The Mixture of Experts (MoE) framework leverages multiple subnetworks as experts (e.g. FNN) with sparse activation, activating only a subset of experts per input to reduce computational costs while maintaining high performance. A milestone in MoE was achieved with Google’s Switch Transformer Fedus et al. 2022 , which introduced a simplified routing mechanism to activate a few experts based on input features. Recent work has emphasized modularity to enhance MoE’s adaptability. For example, RankMean Perin et al. 2024 uses module-level importance scoring to efficiently merge fine-tuned models without access to training data. Similarly, routing mechanisms have gained prominence, with methods like PolyRouter Stripelis et al. 2024 extending the MoE paradigm to multi-LLM querying via predictive routing, and HDMoLE Mu et al. 2024 employing hierarchical routing and dynamic thresholds for multi-domain fine-tuning. Overall, MoE models are centered on

Read the complete text