LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead
Yifan Zhang, Xinkui Zhao, Zuxin Wang, Guanjie Cheng, Yueshen Xu, Shuiguang Deng, Jianwei Yin · 2025
Read LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead by Yifan Zhang, Xinkui Zhao, Zuxin Wang, Guanjie Cheng, Yueshen Xu, Shuiguang Deng, Jianwei Yin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16221.txt.
Introduction
No single language model wins every task, and the best ones charge accordingly. Yifan Zhang and co-authors present LightRouter in 2025, a framework that picks a small team from a larger pool of models and blends their answers, chasing accuracy and thrift together. A selection step finds models that need only a few boot tokens to prove themselves.
Across benchmarks, the routed teams matched or beat standard ensembles, with accuracy gains up to twenty-five percent. Against flagship models, quality held while inference cost fell by up to twenty-seven percent. The system needs no prior knowledge of the models and itself runs on cheap, light components.
Catalog record
| CREATOR | Yifan Zhang, Xinkui Zhao, Zuxin Wang, Guanjie Cheng, Yueshen Xu, Shuiguang Deng, Jianwei Yin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead Yifan Zhang, Xinkui Zhao, Zuxin Wang, Guanjie Cheng, Yueshen Xu, Shuiguang Deng, Jianwei Yin Abstract The rapid advancement of large language models has unlocked remarkable capabilities across a diverse array of natural language processing tasks. However, the considerable differences among available LLMs—in terms of cost, performance, and computational demands—pose significant challenges for users aiming to identify the most suitable model for specific tasks. In this work, we present LightRouter, a novel framework designed to systematically select and integrate a small subset of LLMs from a larger pool, with the objective of jointly optimizing both task performance and cost efficiency. LightRouter leverages an adaptive selection mechanism to identify models that require only a minimal number of boot tokens, thereby reducing costs, and further employs an effective integration strategy to combine their outputs. Extensive experiments across multiple benchmarks demonstrate that LightRouter matches or outperforms widely-used ensemble baselines, achieving up to a 25% improvement in accuracy. Compared with leading high-performing models, LightRouter achieves comparable performance while reducing inference costs by up to 27%. Importantly, our framework operates without any prior knowledge of individual models and relies exclusively on inexpensive, lightweight models. This work introduces a practical approach for efficient LLM selection and provides valuable insights into optimal strategies for model combination. 1 Introduction Recent advances in large language models (LLMs; Zhang et al. (2022) ; Chowdhery et al. (2023) ; Touvron et al. (2023) ; Brown et al. (2020) ; Achiam et al. (2023) ; Liu et al. (2024) ; Guo et al. (2025) ) have demonstrated remarkable capabilities across a wide range of natural language understanding and generation tasks, with different LLMs often excelling in distinct areas and outperforming each other on specific benchmarks or applications ( Bi et al. (2023) ; Ren et al. (2025) ; Zhu et al. (2024) ). In addition to their varied capabilities, LLMs also differ significantly in terms of cost, which adds another layer of complexity to model selection. As the landscape of LLMs continues to expand, users face increasing challenges in identifying the most suitable model for a given task, balancing both performance and cost considerations ( Snell et al. (2024) ; Chen and Varoquaux (2024) ). Moreover, efficiently leveraging the collective expertise of multiple LLMs remains an exciting but largely unresolved direction. This situation raises a fundamental question: How can we effectively select or utilize LLMs to achieve the best possible solution? Existing strategies for model selection generally fall into two categories. The first category is to pre-select a single LLM that is theoretically optimal for a given task, based on prior knowledge, domain intuition, or historical performance statistics ( Feng et al. (2024) ; Xia et al. (2024) ). This approach is appealing for its cost-effectiveness, as it avoids redundant computation by relying on a single model. However, the performance of a single LLM can fluctuate significantly even within the same task, resulting in unstable or suboptimal outputs in practice. Additionally, this method requires continuous monitoring and updating of model knowledge and performance benchmarks, which can be both labor-intensive and impractical as new models and tasks emerge. The second approach involves aggregating the outputs of multiple LLMs using techniques such as majority voting ( Jiang et al. (2023) ; Ravaut et al. (2022) ), averaging, or other ensemble methods ( Wang et al. (2024) ). This strategy often results in improved answer quality and greater robustness, as it leverages the complementary strengths of different models. However, these benefits come at the expense of significantly increased computational resources and latency, since multiple models must be queried for each task. For example, our experiments show that the popular ensemble method MoA ( Wang et al. (2024) ) incurs a computational cost exceeding 600% of that of a single model. Such resource overhead can quickly become prohibitive, particularly in scenarios requiring large-scale deployment. In this work, we seek a more cost-effective and high-performance approach to harnessing the collective capabilities of multiple LLMs. To this end, we introduce LightRouter —a truly out-of-the-box, plug-and-play framework that balances cost and performance without requiring any prior knowledge of tasks, model ability assessments, or additional training. Our design is inspired by modern distributed systems such as Kubernetes, which employ a two-stage scheduling strategy: they first gather lightweight status information from all candidate workloads through efficient probing (e.g., GET requests), and then judiciously allocate resources to the most promising candidates based on this preliminary assessment. Similarly, LightRouter initially queries all candidate models for a small number of boot tokens, thereby quickly and inexpensively assessing the potential of each model without incurring the full computational cost. Based on these initial outputs, LightRouter filters out models that do not meet quality standards and allows only the most promising models to continue generating responses. Finally, the outputs from the remaining models are ensembled to produce the final, optimal answer. We evaluate LightRouter across several popular benchmarks. Extensive experiments demonstrate that LightRouter matches or surpasses leading ensemble methods, achieving up to a 25% improvement in accuracy, and delivers performance on par with state-of-the-art (SOTA), high-cost models while lowering inference costs by 27%. Notably, our framework requires no prior knowledge about individual models and operates efficiently using only inexpensive, lightweight models. Accordingly, our contributions are as follows: • We propose LightRouter, a novel framework for cost-effective and adaptive multi-LLM utilization without requiring prior knowledge of task or model specifics. Unlike static selection or traditional ensembles, LightRouter dynamically queries models and halts underperformers early, minimizing computational overhead. • We design a two-stage adaptive selection mechanism: preliminary partial outputs are used to filter candidates, and only the most promising models proceed to full generation, which significantly reduces unnecessary computation while preserving diversity and robustness. • We introduce an efficient ensembling strategy that aggregates only the top-performing models, achieving a strong balance between accuracy and efficiency and avoiding the high costs of full ensembling. • Extensive experiments on standard LLM benchmarks show that LightRouter achieves higher answer quality and lower inference cost, and further illuminate why adaptive, selective routing is essential for robust multi-model orchestration. 2 Method 2.1 Theoretical Model for LightRouter We introduce the theoretical model for LightRouter, which aims to achieve optimal performance-cost trade-offs by leveraging multiple LLMs in a collaborative manner. 2.1.1 Background Given input X , a generative model produces Y=(y_{1},\ldots,y_{T}) , with P(Y|X)=\prod_{t=1}^{T}P(y_{t}|y_{<t},X) . Early errors in autoregressive generation can propagate, degrading later outputs. We quantify semantic consistency with the ground-truth Y^{\textrm{true}} using S(y_{i},y_{i}^{\textrm{true}})\in[0,1] , and model cumulative error up to step m as \epsilon=1-\prod_{i=1}^{m}S(y_{i},y_{i}^{\textrm{true}}) . Suppose N models \{f_{1},\ldots,f_{N}\} generate outputs Y^{(f_{i})} with sequence-level consistency S^{(f_{i})} . Our objective is to maximize S(Y^{\textrm{final}},Y^{\textrm{true}})-\lambda T_{\textrm{total}} , where T_{\textrm{total}} is total token cost and \lambda is a trade-off parameter. 2.1.2 LightRouter Method Merging Multiple Outputs Given N models {f_{1},\dots,f_{N}} , each generates an output Y^{(f_{i})} . We aggregate a selected set of outputs using a large model f_{\textrm{merge}} as Y^{\textrm{final}}=f_{\textrm{merge}}({Y^{(f_{i})}:f_{i}\in\mathcal{S}}) , where \mathcal{S} is a subset of candidate models. If each S^{(f_{i})} is an independent random variable with mean \mu and variance \sigma^{2} , then merging k outputs by f_{\textrm{merge}} reduces variance to \sigma^{2}/k . This process leverages the collective wisdom of multiple models, leading to higher and