Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning
Haozhen ZhangTao FengJiaxuan YouUniversity of Illinois at Urbana-Champaign, {haozhenz, taofeng2, jiaxuan}@illinois.edu, [email protected] · 2025
Read Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning by Haozhen ZhangTao FengJiaxuan YouUniversity of Illinois at Urbana-Champaign, {haozhenz, taofeng2, jiaxuan}@illinois.edu, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09033.txt.
Introduction
Haozhen Zhang, Tao Feng, and Jiaxuan You present Router-R1, a framework that teaches a language model to route questions among other language models. Most existing routers assign each query to a single model in one round. That one-to-one mapping cannot combine the complementary strengths of several models on hard tasks.
Router-R1 instead treats multi-model routing and aggregation as a sequential decision process learned through reinforcement learning. The router is itself a capable language model that interleaves internal thinking with calls to other models, folding each response into its evolving context. A light rule-based reward combines format, final outcome, and a cost term that balances performance against price. Because the router conditions only on simple model descriptors such as pricing and latency, it generalizes to unseen models. On seven general and multi-hop question-answering benchmarks, it outperformed several strong baselines.
Catalog record
| CREATOR | Haozhen ZhangTao FengJiaxuan YouUniversity of Illinois at Urbana-Champaign, {haozhenz, taofeng2, jiaxuan}@illinois.edu, [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09033
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning Haozhen ZhangTao FengJiaxuan YouUniversity of Illinois at Urbana-Champaign, {haozhenz, taofeng2, jiaxuan}@illinois.edu, [email protected] Abstract The rapid emergence of diverse large language models (LLMs) has spurred the development of LLM routers that assign user queries to the most suitable model. However, existing LLM routers typically perform a single-round, one-to-one mapping ( i.e. , assigning each query to a single model in isolation), which limits their capability to tackle complex tasks that demand the complementary strengths of multiple LLMs. In this paper, we present Router-R1 , a reinforcement learning (RL)-based framework that formulates multi-LLM routing and aggregation as a sequential decision process. Router-R1 instantiates the router itself as a capable LLM, leveraging its reasoning ability to interleave “think” actions (internal deliberation) with “route” actions (dynamic model invocation), and integrates each response into its evolving context. To facilitate learning, we employ a lightweight rule-based reward comprising format rewards, final outcome rewards, and a novel cost reward for optimizing the balance between performance and cost, opening a pathway toward enhancing performance-cost trade-offs via RL. Router-R1 also conditions only on simple model descriptors such as pricing, latency, and example performance, enabling strong generalization to unseen model selection. Experiments on seven general and multi-hop QA benchmarks show that Router-R1 outperforms several strong baselines, achieving superior performance while maintaining robust generalization and cost management. ulab-uiuc/Router-R1 Hugging Face Collection 1 Introduction Large language models (LLMs) have proliferated at an unprecedented pace, with new architectures and fine-tuned variants released on a monthly or even weekly basis ( naveed2023comprehensive ; chang2024llmsurvey ) . To harness the complementary strengths of multiple LLMs ( e.g. , one model’s fluency versus another’s factual accuracy), LLM routers have emerged as a critical infrastructure component, dynamically dispatching user queries to a single selected model to maximize answer quality and efficiency ( chen2024routerdc ; feng2024graphrouter ; chen2023frugalgpt ; ding2024hybrid ; vsakota2024fly ; stripelis2024tensoropera ; dai2024C2MAB-V ) . While this one-shot routing strategy can improve average performance, it overlooks the fact that truly complex reasoning tasks often require coordinated interactions among multiple models, orchestrating not just a single choice but a sequence of model calls to leverage their complementary strengths. This observation raises a key challenge: how can we coordinate multiple LLMs in a multi-round routing and aggregation process to jointly solve complex tasks? Answering this challenge is non-trivial. First , the discrete decision process of selecting which LLM to call at each round is inherently non-differentiable, precluding straightforward end-to-end training via backpropagation. Although prior work has applied gradient-based methods to single-shot routing ( chen2024routerdc ; feng2024graphrouter ; chen2023frugalgpt ; ding2024hybrid ) , extending them to multi-round selection and aggregation rapidly becomes intractable. Second , existing routers operate in a single-step regime: given a query, pick one model, collect its output, and stop ( chen2024routerdc ; feng2024graphrouter ) . However, complex tasks ( e.g. , multi-hop question answering) demand a sequence of interleaved reasoning and model-selection decisions, and a one-off choice rarely suffices. Therefore, we must design an interplay mechanism that alternates between “long thought” reasoning (internal deliberation) and targeted LLM selection to refine an answer iteratively. To address these challenges, we introduce Router-R1 , a reinforcement learning–based framework for multi-round LLM routing and aggregation. Rather than making a single dispatch decision, we formulate LLM coordination as a sequential decision-making problem. At each step, Router-R1 chooses whether to perform internal reasoning (“think”) or to invoke a specific model from a pool of available LLMs (“route”), gradually constructing an answer through iterative interaction. To support this interplay between reasoning and routing, we instantiate the router itself as a capable LLM, leveraging its inherent reasoning capability to perform long-form deliberation and targeted model selection. This flexible reasoning and model selection interleaving enables Router-R1 to adaptively compose the strengths of multiple LLMs in a task-aware manner. To optimize this decision policy, we adopt reinforcement learning and design a simple yet effective rule-based reward function composed of three parts: a format reward for producing well-structured outputs, a final outcome reward based on task correctness, and a cost reward that penalizes excessive use of expensive routed models, providing Router-R1 the capability to navigate performance–cost trade-offs during training. Additionally, Router-R1 exhibits strong generalization capability to newly added LLM candidates without the need for retraining by conditioning its routing decisions on simple descriptors such as pricing, latency, and example performance. Together, these components make Router-R1 a robust and flexible solution for coordinating multiple LLMs to solve complex reasoning tasks. Through comprehensive experiments on seven diverse QA benchmarks, covering both general and multi-hop question answering, we demonstrate that Router-R1 consistently outperforms several strong baselines and achieves state-of-the-art performance. Further analyses on cost-aware routing and generalization to unseen LLMs also highlight the flexibility and robustness of our approach. Our contributions are summarized as follows: • We propose Router-R1 , a reinforcement learning–based framework for multi-round LLM routing and aggregation. By instantiating the router itself as a capable LLM, Router-R1 naturally interleaves internal reasoning and external model selection, enabling adaptive coordination across multiple LLMs for complex task solving. • We design a simple and effective rule-based reward function consisting of format rewards, final outcome rewards, and cost rewards, enabling Router-R1 to navigate performance–cost trade-offs. Additionally, by conditioning on simple model descriptors, Router-R1 can generalize to unseen LLMs without retraining. • Through extensive experiments on seven question-answering benchmarks, we show that Router-R1 outperforms several competitive baselines, achieving superior performance and robust generalization. 2 Related Work 2.1 Query-based Routers for LLM Selection The rapid rise of various LLMs has spurred the development of query-based LLM routers, which aim to direct queries to the most appropriate model to improve response quality and efficiency. HybridLLM ( ding2024hybrid ) proposes a dynamic router that selects between small and large LLMs based on predicted query difficulty and a user-defined quality budget. GraphRouter ( feng2024graphrouter ) frames LLM selection as inductive edge prediction over a task–query–model graph, enabling cost-performance estimation and effortless integration of new LLMs. To explicitly balance performance and cost, FrugalGPT ( chen2023frugalgpt ) adopts an LLM cascade approach, while FORC ( vsakota2024fly ) routes queries to appropriately sized models for cost-effective inference. TO-Router ( stripelis2024tensoropera ) unifies multiple domain-specific expert LLMs under one interface and dispatches queries based on task needs. C2MAB-V ( dai2024C2MAB-V ) leverages a cost-aware combinatorial multi-armed bandit to dynamically select optimal LLM subsets. RouterDC ( chen2024routerdc ) enhances routing via dual contrastive learning between queries and LLM embeddings. Finally, RouteLLM ( ong2024routellm ) utilizes human preference data to dynamically choose between strong and weak LLMs, effectively reducing costs while maintaining quality. In contrast to previous methods, Router-R1 treats routing as a sequential decision process, interleaving internal “think” steps with multi-round model routing to refine its answer by instantiating the router itself as a capable LLM. Moreover, its RL-based training harnesses a cost reward to balance performance and cost, enabling flexible performance–cost trade-offs and resource-aware routing. 2.2 Optimizing LLM Behaviors via Reinforcement Learning In recent years, reinforcement learning (RL) has emerged as a powerful paradigm for fine-tuning large language models (LLMs) to better align with human preferences. Early work like RLHF (Reinforcement Learning from Human Feedback) ( RLHF ) , trains a reward model on human judgments and then