Leveraging Uncertainty Estimation for Efficient LLM Routing

Tuo Zhang, Asal Mehradfar, Dimitrios Dimitriadis, Salman Avestimehr · 2025

Read Leveraging Uncertainty Estimation for Efficient LLM Routing by Tuo Zhang, Asal Mehradfar, Dimitrios Dimitriadis, Salman Avestimehr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11021.txt.

Introduction

Tuo Zhang, Asal Mehradfar, Dimitrios Dimitriadis, and Salman Avestimehr route each request to the right language model by reading the model's own uncertainty. Big models cost more per answer, and small ones fail quietly on hard prompts. Routers usually lean on human preference data or fixed benchmarks.

Their router estimates confidence and sends hard cases up, easy cases down. Quality is judged not only by accuracy but by a model-judge that mimics human ratings. Tests span cost and quality together, which earlier work rarely did. Edge devices and cloud servers share the load in their setup. Smarter routing cuts bills without dulling answers. Uncertainty, long a diagnostic, becomes a traffic signal.

Catalog record

CREATORTuo Zhang, Asal Mehradfar, Dimitrios Dimitriadis, Salman Avestimehr
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11021

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Uncertainty Estimation for Efficient LLM Routing Tuo Zhang, Asal Mehradfar, Dimitrios Dimitriadis, Salman Avestimehr Abstract Deploying large language models (LLMs) in edge-cloud environments requires an efficient routing strategy to balance cost and response quality. Traditional approaches prioritize either human-preference data or accuracy metrics from benchmark datasets as routing criteria, but these methods suffer from rigidity and subjectivity. Moreover, existing routing frameworks primarily focus on accuracy and cost, neglecting response quality from a human preference perspective. In this work, we propose the Confidence-Driven LLM Router, a novel framework that leverages uncertainty estimation to optimize routing decisions. To comprehensively assess routing performance, we evaluate both system cost efficiency and response quality. In particular, we introduce the novel use of LLM-as-a-Judge to simulate human rating preferences, providing the first systematic assessment of response quality across different routing strategies. Extensive experiments on MT-Bench, GSM8K, and MMLU demonstrate that our approach outperforms state-of-the-art routing methods, achieving superior response quality while maintaining cost efficiency. 1 Introduction The deployment of AI models on edge devices is increasingly following a hybrid approach, where small language models (SLMs) run on-device (e.g., smartphones and IoT devices) while larger, more powerful models remain in the cloud Apple (2024) . This setup provides a balance between efficiency and performance, allowing low-latency responses for simple queries while reserving cloud-based LLMs for more complex tasks. However, determining when to offload queries to the cloud is a crucial challenge: calling the cloud unnecessarily increases cost and latency, whereas over-relying on local SLMs risks suboptimal response quality. An effective routing strategy is essential to dynamically balance cost and performance. Traditional cascading routers, which sequentially query models until a satisfactory response is obtained Chen et al. (2023) , are inefficient for edge-cloud settings due to latency and redundant model calls. Recent predictive routing approaches aim to preemptively select the best model for a given query, with two leading solutions: TO-Router Stripelis et al. (2024) , which trains router on accuracy-based benchmarks and RouteLLM Ong et al. (2024) , which relies on human preference selection for router training. Figure 1: Performance of human preference-based router with varying training sample sizes. Routing efficiency even becomes worse as the number of training samples increases, indicating that additional data does not necessarily improve performance. While human preference data and benchmark accuracy are commonly used as performance indicators in router training, our results reveal two major limitations that hinder data efficiency and system utilization. First, human judgment is not always reliable. User ratings are subjective and inconsistent, often failing to provide an accurate ranking of model performance. Also, collecting human-preference data is resource-intensive, requiring manual evaluation of each sample on a case-by-case basis. These issues are particularly evident in the arena dataset Chiang et al. (2024) , where the distribution of preference data across models is sparse and uneven, complicating router training. Empirically, we demonstrate that arena data does not follow a scaling law to validate our argument. As shown in Figure 1 , increasing the dataset size does not necessarily improve routing performance. Instead, adding more data can introduce noise and inconsistencies, potentially degrading routing accuracy rather than enhancing it. Second, accuracy is an incomplete indicator. Binary accuracy metrics do not capture response confidence, meaning two models may provide correct answers, yet one is significantly more precise and reliable. To address the limitations of state-of-the-art methods, we propose the Confidence-Driven LLM Router System , which leverages Semantic Entropy (SE) as an uncertainty measure to guide routing decisions. Instead of relying on human preferences or accuracy-based thresholds, our system uses semantic entropy to measure model confidence. This enables the router to offload queries to cloud-based LLMs when higher certainty is needed, which keeps confident responses on-device to minimize cost. As the motivational example shown in Figure 2 , by adopting uncertainty as a routing signal, our approach dynamically optimizes response quality and computational efficiency, making it better suited for real-world edge-cloud deployments. Figure 2: Routing performance/cost trade-off between strong model (GPT-4) and weak model (Mixtral-8x7B). All routers shown, except the random router, use the same kNN-based model architecture. 2 Method 2.1 Overview Inspired by prior work Kuhn et al. (2023) ; Bakman et al. (2024) , we quantify uncertainty in natural language generation based on semantic content rather than token-level variations, using it as a performance indicator for training the router. SE Kuhn et al. (2023) captures uncertainty by clustering generated outputs with equivalent meanings and computing entropy over their aggregated probabilities. Unlike traditional entropy, which treats all token sequences distinctly, SE accounts for semantic equivalence, ensuring paraphrases contribute to the same uncertainty estimate. A lower SE score indicates higher confidence, while a higher score signals greater uncertainty. Formally, we define the probability of a meaning cluster c given an input prompt \mathbf{x} as: p(c|\mathbf{x})=\sum_{\begin{subarray}{c}s,\mathbf{x}\in c\end{subarray}}p(s|\mathbf{x}). (1) The semantic entropy of \mathbf{x} is then computed as: SE(\mathbf{x})=-\frac{1}{|C|}\sum_{i=1}^{|C|}\log p(C_{i}|\mathbf{x}), (2) where C represents the set of all clusters. 2.2 System Design of the Confidence-Driven LLM Router The training and deployment of the Confidence-Driven LLM Router consist of three key phases: Phase 1: Router Data Preparation. To create a training dataset that reflects real deployment scenarios, we select factual-related datasets, such as Natural QA and Trivia QA, to probe model confidence and knowledge capabilities. Additionally, domain-specific instruction datasets can be incorporated to tailor the router to specialized applications. Clustering generated outputs is a critical preprocessing step before computing SE. In this work, we adopt a bidirectional entailment mechanism following the previous work Kuhn et al. (2023) . The first generated response initializes a cluster. For each subsequent response, a semantic entailment classifier, fine-tuned on DeBERTa-large model He et al. (2020) , evaluates bidirectional entailment between the response and the current cluster representative. If both forward and backward entailments hold, the response is assigned to the existing cluster; otherwise, a new cluster is formed. This bidirectional criterion ensures that only semantically equivalent responses are grouped, allowing the number of clusters to be dynamically determined based on meaning variations among generated outputs. Phase 2: Constructing Preference Data from Semantic Entropy Scores. In the second phase, we create preference data by comparing the SE scores across models for each unique prompt. Although no two SE values are exactly the same, some prompts yield similar performance between models, which we denote as a “tie” case. To identify ties, we treat cases with close uncertainty levels as equivalent, where neither model is a clear winner. To quantify the relative difference in uncertainty, we compute the normalized SE difference between the two models as: \delta_{\text{SE}}(\mathbf{x})=\left|\frac{SE_{\text{strong}}(\mathbf{x})-SE_{\text{weak}}(\mathbf{x})}{SE_{\text{strong}}(\mathbf{x})}\right| (3) SE_{\text{strong}}(\mathbf{x}) represents the high-cost model, and SE_{\text{weak}}(\mathbf{x}) represents the low-cost model. Using this metric, we determine the preferred model as: \text{Winner}=\begin{cases}\arg\min\limits_{M}SE(M,\mathbf{x})&\text{if }\delta_{\text{SE}}(\mathbf{x})>\tau,\\ \text{Tie}&\text{otherwise}.\end{cases} (4) The predefined threshold \tau controls sensitivity to uncertainty differences. If \delta_{\text{SE}}(\mathbf{x}) exceeds the predefined \tau , the uncertainties are considered sufficiently distinct, and the model with the lower semantic entropy is designated as the preferred model. Otherwise, we consider both models equal in performance for the given prompt. A higher \tau enforces stricter distinctions, aligning the preference data more closely with traditional accuracy-based evaluations. Conversely, a lower \tau increases sensitivity to subtle linguistic variations in model responses. By tuning \tau , we balance robustness with linguistic granularity. Phase 3:

Read the complete text