Guarded Query Routing for Large Language Models

Richard Šléher, William Brach, Tibor Sloboda, Kristián Košťál, Lukas Galke · 2025

Read Guarded Query Routing for Large Language Models by Richard Šléher, William Brach, Tibor Sloboda, Kristián Košťál, Lukas Galke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14524.txt.

Introduction

Sending each user question to the right language model is a sorting chore with a safety twist: stray, foreign, or unsafe queries must be caught, not misrouted. Richard Šléher and co-authors release in 2025 a benchmark for guarded query routing across law, finance, and healthcare, with seven extra datasets of out-of-domain traps. They race model-based routers, guardrail systems, and plain classifiers.

A simple wide network with out-of-domain detection won the balance: eighty-eight percent accuracy in under four milliseconds. A classic embedding method was fastest, under a millisecond, at eighty percent. Big models scored highest, ninety-one percent, but answered hundreds of times slower.

Catalog record

CREATORRichard Šléher, William Brach, Tibor Sloboda, Kristián Košťál, Lukas Galke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14524

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Guarded Query Routing for Large Language Models Richard Šléher, William Brach, Tibor Sloboda, Kristián Košťál, Lukas Galke Abstract Query routing, the task to route user queries to different large language model (LLM) endpoints, can be considered as a text classification problem. However, out-of-distribution queries must be handled properly, as those could be about unrelated domains, queries in other languages, or even contain unsafe text. Here, we thus study a guarded query routing problem, for which we first introduce the Guarded Query Routing Benchmark (GQR-Bench, released as Python package gqr), covers three exemplary target domains (law, finance, and healthcare), and seven datasets to test robustness against out-of-distribution queries. We then use GQR-Bench to contrast the effectiveness and efficiency of LLM-based routing mechanisms (GPT-4o-mini, Llama-3.2-3B, and Llama-3.1-8B), standard LLM-based guardrail approaches (LlamaGuard and NVIDIA NeMo Guardrails), continuous bag-of-words classifiers (WideMLP, fastText), and traditional machine learning models (SVM, XGBoost). Our results show that WideMLP, enhanced with out-of-domain detection capabilities, yields the best trade-off between accuracy (88%) and speed (¡4ms). The embedding-based fastText excels at speed (¡1ms) with acceptable accuracy (80%), whereas LLMs yield the highest accuracy (91%) but are comparatively slow (62ms for local Llama-3.1:8B and 669ms for remote GPT-4o-mini calls). Our findings challenge the automatic reliance on LLMs for (guarded) query routing and provide concrete recommendations for practical applications. Source code is available: https://github.com/williambrach/gqr . † † paperid: 1354 1 Introduction Large Language Models (LLMs) are responding to millions of queries each day. To save computational resources, it is of high interest to route each query to LLMs of appropriate sizes, inference compute budgets, and domain specialization. We view this query routing problem as an instance of text classification, where the task is to classify a query into a set of suitable endpoints (e.g., LLM agents, rule-based chatbots, human chat partners). See Figure 1 . While such a routing mechanism can be prepared for in-distribution queries (e.g., known languages, a pre-defined set of target domains, generally safe inputs), it must also deal with out-of-distribution queries, such as queries in other domains, languages, or even unsafe inputs. Figure 1: A guarded query routing system that efficiently filters out-of-domain (OOD) queries while directing valid in-domain (ID) queries to appropriate domain-specific LLMs. We denote this task as guarded query routing. Guarded query routing is an extremely challenging task because the models cannot be prepared for all possible out-of-distribution queries. This implies that we must assume that there is no training data available for out-of-distribution queries (otherwise, they would not be out of distribution). Presumably for this reason, practitioners currently rely on LLMs to carry out the routing, such as NVIDIA NeMo Guardrails [ Rebedea et al.(2023)Rebedea, Dinu, Sreedhar, Parisien, and Cohen ] or LlamaGuard [ Inan et al.(2023)Inan, Upasani, Chi, Rungta, Iyer, Mao, Tontchev, Hu, Fuller, Testuggine, and Khabsa ] . However, LLM calls come with extensive computational costs, and we hypothesize that efficient text classifiers combined with an out-of-distribution detection mechanism would perform similarly well with less compute. To test our hypothesis, we first set up a new evaluation benchmark, called Guarded Query Routing Benchmark (GQR-Bench), that simulates a guarded query routing problem. GQR-Bench is composed of three datasets that we consider our target domains (law, finance, health), as well as seven datasets from which we draw different types of out-of-distribution queries: out-of-domain queries, queries in different languages, and queries with unsafe inputs. We further provide a suggested evaluation methodology that balances in-domain accuracy and robustness against out-of-distribution queries. Based on recent surveys in text classification [ Galke et al.(2025)Galke, Scherp, Diera, Karl, Lin, Khera, Meuser, and Singhal ] , we then evaluate strong text classification methods such as BERT [ Devlin et al.(2018)Devlin, Chang, Lee, and Toutanova ] , fastText [ Joulin et al.(2017)Joulin, Grave, Bojanowski, and Mikolov ] , and WideMLP [ Galke and Scherp(2022) ] on GQR-Bench, testing their suitability as guarded query routers. We also test traditional text classification methods (e.g., SVM, XGBoost) on top of contextualized embeddings from a pre-trained language model. We test these methods against common strategies for standard query routing (e.g., routing via a strong LLM such as GPT-4o-mini) as well as standard guardrails to prevent unsafe inputs (e.g., NVIDIA NeMO Guardrails), which we consider as our baselines. We show that efficient text classifiers can rival costly LLM-based routers for guarded query routing. WideMLP achieves 95% of the LLM’s relative performance while being orders of magnitude faster (¡1ms latency) and eliminating API costs. We also demonstrate that standard guardrails and prior routing methods fail on our GQR-Bench, underscoring the task’s difficulty. This work challenges the default reliance on LLMs for routing, offering a practical, high-performance alternative. In summary, the main contributions of this paper are: 1. We present a new benchmark, GQR-Bench is released as a Python package ( https://pypi.org/project/gqr/ ), along with an evaluation methodology, for guarded query routing, where the routing mechanisms need to determine the correct domain, while being robust against out-of-distribution queries. 2. We run and evaluate several models on this benchmark, including large language models, efficient text classifiers, and traditional machine learning models based on embeddings – to test their suitability for guarded query routing. 3. Our results show that efficient text classifiers display good performance with notable improvements in latency, challenging common practice to employ LLMs for the task of query routing. 2 Related Work The rationale behind query routing lies in its ability to improve cost efficiency while maintaining the integrity of responses [ Shafran et al.(2025)Shafran, Schuster, Ristenpart, and Shmatikov ] . This allows users to optimise the allocation of computational resources and operational expenses by directing less complex queries to smaller, more economical models, reserving more sophisticated models for tasks requiring advanced capabilities or rejecting queries identified as malicious or outside the domain of interest [ Hu et al.(2024)Hu, Bieker, Li, Jiang, Keigwin, Ranganath, Keutzer, and Upadhyay , Huang et al.(2025)Huang, Ling, Liang, Lin, Chen, Zhong, Wu, and Lin , Shnitzer et al.(2023)Shnitzer, Ou, Silva, Soule, Sun, Solomon, Thompson, and Yurochkin ] . A primary application domain involves the systematic routing between models with different sizes and functional capabilities, often using contrastive learning to differentiate between the strengths of various models [ Ong et al.(2025)Ong, Almahairi, Wu, Chiang, Wu, Gonzalez, Kadous, and Stoica , Chen et al.(2024)Chen, Jiang, Lin, Kwok, and Zhang ] . This dynamic allocation methodology enables systems to delegate less cognitively demanding tasks to models with fewer parameters, thereby conserving the computational resources of larger architectures for queries that require enhanced precision or advanced reasoning capabilities. Studies using real-world examples show that this query routing can reduce the use of large models by up to 40% without affecting their quality [ Varangot-Reille et al.(2025)Varangot-Reille, Bouvard, Gourru, Ciancone, Schaeffer, and Jacquenet , Ding et al.(2024)Ding, Mallick, Wang, Sim, Mukherjee, Ruhle, Lakshmanan, and Awadallah ] . The theoretical foundations of query routing extend beyond mere model selection. It takes inspiration from conceptually adjacent frameworks, such as Mixture-of-Experts architectures [ Lee et al.(2025)Lee, Soldaini, Cohan, Seo, and Lo , Shazeer et al.(2017)Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton, and Dean ] , which incorporate internal routing mechanisms, and Retrieval-Augmented Generation (RAG) systems. RAG systems employ contextual classification 1 1 1 https://weaviate.io/developers/contributor-guide/contextionary/classification-benchmarks methods to select appropriate document repositories and

Read the complete text