Unsupervised Query Routing for Retrieval Augmented Generation
Feiteng Mu, Liwen Zhang, Yong Jiang, Zhen Zhang, Wenjie Li, Pengjun Xie, Fei Huang · 2025
Read Unsupervised Query Routing for Retrieval Augmented Generation by Feiteng Mu, Liwen Zhang, Yong Jiang, Zhen Zhang, Wenjie Li, Pengjun Xie, Fei Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-07793.txt.
Introduction
Mu and colleagues wrote in twenty twenty-five about routing questions to the best search engine without labeled training data. Retrieval-augmented systems answer by first searching outside sources. Picking the right engine for each query improves answers. Past routing methods need hand-labeled data, which costs much and transfers poorly. The team instead builds a best-possible reference answer for each query. They judge each engine by how close its aided answer gets.
Those judgments create training data on their own, with no manual labels. The method can process large sets of real user queries at scale. Tests across five data sets showed strong gains. Scalability and performance on new kinds of queries both improved. The approach suits fast-changing search settings.
Catalog record
| CREATOR | Feiteng Mu, Liwen Zhang, Yong Jiang, Zhen Zhang, Wenjie Li, Pengjun Xie, Fei Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-07793
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unsupervised Query Routing for Retrieval Augmented Generation Feiteng Mu, Liwen Zhang, Yong Jiang, Zhen Zhang, Wenjie Li, Pengjun Xie, Fei Huang Abstract Query routing for retrieval-augmented generation aims to assign an input query to the most suitable search engine. Existing works rely heavily on supervised datasets that require extensive manual annotation, resulting in high costs and limited scalability, as well as poor generalization to out-of-distribution scenarios. To address these challenges, we introduce a novel unsupervised method that constructs the "upper-bound" response to evaluate the quality of retrieval-augmented responses. This evaluation enables the decision of the most suitable search engine for a given query. By eliminating manual annotations, our approach can automatically process large-scale real user queries and create training data. We conduct extensive experiments across five datasets, demonstrating that our method significantly enhances scalability and generalization capabilities. 1 Introduction Retrieval Augmented Generation (RAG) Lewis et al. 2020 ; Gao et al. 2023 typically begins with a retrieval phase where user queries are scoured through search engines like Google and Bing to gather relevant background documents before engaging large language models (LLMs) OpenAI 2023 ; Qwen Team 2023 . In today’s digital landscape, even among major comprehensive search engines, each has its strengths and weaknesses, offering unique features and capabilities. Consequently, Query Routing Sugiura and Etzioni 2000 ; Wang et al. 2024 ; Mu et al. 2024 seeks to direct queries to the most suitable search engines to optimize performance and reduce costs. Current query routing methods Shnitzer et al. 2023 ; Mu et al. 2024 primarily use open-sourced (query, answer) paired data to create training labels. Specifically, by leveraging these gold-standard answers, they assess the quality of retrieval-augmented responses for queries routed through different search engines, thereby identifying the most effective engine for each query. However, the reliance on annotated paired data presents challenges such as limited diversity and high annotation costs, which significantly restrict scalability. Moreover, there exists a pronounced discrepancy between the distribution of public datasets and real user queries , which severely impacts the out-of-distribution generalization Shnitzer et al. 2023 of these methods, limiting their effectiveness in real-world applications. To enable models to generalize effectively to actual user queries, it is essential to train them directly on real user queries. Nevertheless, real user queries often lack gold-standard answers, presenting a major challenge in determining the appropriate search engine assignment, particularly when no definitive answer is available . Search Strategies Qwen2-max GPT4 No-RAG 2.189 2.014 Using one search engine Quark 2.651 2.456 Bing 2.615 2.390 Google 2.610 2.420 Using two search engines Quark+Bing 2.689 2.495 Quark+Google 2.701 2.486 Bing+Google 2.674 2.478 Using three search engines Quark+Bing+Google 2.727 2.521 Table 1: RAG results across various search strategies. We merge all test sets and calculate average end2end QA scores, i.e., Correctness, on the merged examples. No-RAG denotes that LLMs respond to the query without retrieval. Scores with bold denote the best results. To address this challenge, we must transform our problem into a labelling-free form. Since query routing fundamentally involves identifying the optimal assignment within the limited computing resources, such as inference time and budget, we can envision an ideal situation in which we possess infinite computing resources. In this case, an intuitive and obvious strategy is to search a query across all search engines, then merge multi-sourced documents for RAG 1 1 1 For simplicity, we denote the generated response when retrieving from multiple tools as the multi-sourced response. Correspondingly, we denote the generated response when retrieving from only one tool as the single-sourced response. . Due to the complementarity of different search engines Mu et al. 2024 , the merged multi-sourced documents should have a better recall than the single-sourced documents. That is, multi-sourced responses are likely to exhibit higher quality than single-source responses. To support this assertion, we calculate the RAG results across various search strategies and different LLMs, as illustrated in Table 1 . We find that utilizing more search engines consistently leads to improved results. This trend is observed across multiple LLMs. Based on this prior knowledge, we can view the multi-sourced responses as an upper bound for unsupervised query routing. Now, let us return to the practical situation of limited computing resources. To evaluate the quality of a single-sourced response, we can compare it to the corresponding multi-sourced response, which serves as an upper bound. By assessing the similarity between the single-sourced response and its upper bound, we are able to determine its quality. If the single-sourced response shows a high level of similarity to its upper bound, this suggests that the query should be assigned to the corresponding search engine. This chain of thought motivates us to develop an unsupervised query routing method that directly leverages real user data to generate training labels in an automated manner, eliminating the need for manual annotation. In this work, we propose a novel unsupervised query routing method that directly creates training data without human annotations. Our method mainly consists of four steps. In the first step, we search every user query in only a single search engine to build single-sourced responses. In the second step, we search every user query across all available search engines to establish multi-sourced responses, which are used as the upper bound for unsupervised query routing. Next, given a query and its corresponding single-sourced responses, we use several automated metrics to evaluate the quality of the single-sourced responses, allowing us to determine which search engine best fits the query. Finally, by constructing training data automatically, we train our routing model. Our method operates without the need for manual annotation, enabling the automation of large-scale processing of real-world user queries and facilitating the construction of training data. Consequently, our approach can be effectively scaled and has a strong generalization capability for real user queries. In summary, our contributions are as follows. (1) We introduce a novel unsupervised query routing method capable of automatically processing large-scale, unlabeled real user data for model training. This simple yet effective approach significantly reduces annotation costs and offers excellent scalability potential. (2) As far as we know, we are the first to explore unsupervised query routing within the RAG scenario. Our method serves as a universal framework, holding significant promise for providing valuable insights into the field of tool learning. (3) We validate our method across five datasets in a non-i.i.d. setting. The experiments demonstrate that our approach exhibits i) outstanding generalization ability, ii) excellent scalability, and iii) consistent effectiveness across multiple LLMs. 2 Related Work Query Routing for LLMs Nearly monthly released LLMs Qwen Team 2023 ; Dubey et al. 2024 are trained with various data, resulting in a variety of strengths and weaknesses in versatile downstream tasks Jiang et al. 2023 . Therefore, researchers Narayanan Hari and Thomson 2023 ; Lu et al. 2023 ; Liu et al. 2024 ; Shnitzer et al. 2023 ; Chen et al. 2023b ; Šakota et al. 2024 are devoted to learning the optimal assignment of queries to different LLMs. The basic idea is to leverage the complementarity of various LLMs to achieve superior performance compared to any single model across a range of tasks. Notably, these methods typically require substantial amounts of annotated training data, leading to high costs