Causal LLM Routing: End-to-End Regret Minimization from Observational Data

Asterios Tsiourvas, Wei Sun, Georgia Perakis · 2025

Read Causal LLM Routing: End-to-End Regret Minimization from Observational Data by Asterios Tsiourvas, Wei Sun, Georgia Perakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16037.txt.

Introduction

Asterios Tsiourvas, Wei Sun, and Georgia Perakis build in 2025 a causal routing method that picks which language model should answer each query, learning from logged outcomes alone. Routing balances accuracy against cost across a model pool. Past methods predict scores first and pick second, which stacks errors and needs costly full evaluations.

Observational logs record only the deployed model's result, and the method learns despite that gap. Two surrogate goals make regret minimization tractable, one a classification bound and one a softmax-weighted form that recovers the best policy. An interval-conditioned variant handles varied cost tastes. Benchmark tests beat prior routers across embedding choices. The frame suits live systems where trying every model is impossible.

Catalog record

CREATORAsterios Tsiourvas, Wei Sun, Georgia Perakis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16037

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Causal LLM Routing: End-to-End Regret Minimization from Observational Data Asterios Tsiourvas, Wei Sun, Georgia Perakis Abstract LLM routing aims to select the most appropriate model for each query, balancing competing performance metrics such as accuracy and cost across a pool of language models. Prior approaches typically adopt a decoupled strategy, where metrics are first predicted and the model is then selected based on these estimates. This setup is prone to compounding errors and often relies on full-feedback data, where each query is evaluated by all candidate models, which is costly to obtain and maintain in practice. In contrast, we learn from observational data , which records only the outcome of the model actually deployed. We propose a causal end-to-end framework that learns routing policies by minimizing decision-making regret from observational data. To enable efficient optimization, we introduce two theoretically grounded surrogate objectives: a classification-based upper bound, and a softmax-weighted regret approximation shown to recover the optimal policy at convergence. We further extend our framework to handle heterogeneous cost preferences via an interval-conditioned architecture. Experiments on public benchmarks show that our method outperforms existing baselines, achieving state-of-the-art performance across different embedding models. 1 Introduction LLM routing is an emerging research area focused on optimizing model selection for each input query, balancing performance and cost across a pool of available LLMs. Because LLM performance varies significantly by task and input ( Hu et al., 2024 ) , as well as computational cost ( Ong et al., 2024 ) , dynamic routing strategies have been proposed to select the most suitable model per query ( Shnitzer et al., 2023 ) . This challenge becomes even more critical in agentic applications, where multiple LLM calls may be made within a single workflow, making efficient model selection essential for user experience and resource allocation. As LLM deployment scales, routing also contributes to environmental sustainability by reducing unnecessary computation ( Singh et al., 2025 ) . Routing methods can be broadly classified based on whether they invoke one or multiple models per query. Multi-model approaches include non-predictive routing , which cascades models sequentially from light to heavy ( Wang et al., 2023 ; Chen et al., 2023 ) , and predictive ensembles , which learn to combine outputs or scores from multiple LLMs ( Shekhar et al., 2024 ; Huang et al., 2025 ) . While these methods can improve accuracy and robustness, they incur significant computational cost and latency due to repeated inference. By contrast, predictive routing methods, including our approach, select a single LLM for each query by training a router that maps input queries to the most appropriate model ( Shnitzer et al., 2023 ; Ong et al., 2024 ; Somerstep et al., 2025 ) . This framework offers a more scalable and cost-efficient solution, particularly in settings where minimizing latency and compute resources is critical. A common formulation in predictive routing is to maximize a utility function of the form y_{x}(t)=a_{x}(t)-\lambda\cdot c_{x}(t) , where a_{x}(t) and c_{x}(t) denote the quality (e.g., accuracy) and cost for model t on a given query x , and \lambda\geq 0 captures user cost sensitivity, or willingness-to-pay. Existing methods typically adopt a decoupled approach: separate predictors are trained for each metric, and routing is performed by selecting the model with the highest estimated utility. However, decision quality is highly sensitive to these predictors, and errors can compound, especially when incorporating additional metrics (e.g., latency, faithfulness, alignment), increasing complexity and uncertainty. Another fundamental limitation in the existing literature on predictive routing is its reliance on full-feedback datasets. Prior work ( Ong et al., 2024 ) ( Somerstep et al., 2025 ) assumes access to data where each query has been evaluated by all available LLMs. This assumption is impractical: (i) the computational and monetary cost of exhaustively querying all models is prohibitive; and (ii) the rapid pace of LLM development makes it challenging to maintain comprehensive, up-to-date evaluation datasets. In contrast, observational data , where each query is evaluated by only one model, is readily available from real-world LLM deployments, making it far more scalable than full feedback datasets. However, it introduces treatment bias from historical routing policies, which can lead to suboptimal decisions if not properly addressed ( Swaminathan and Joachims, 2015 ; Künzel et al., 2019 ) . To the best of our knowledge, this is the first work to (i) learn LLM routing from observational data and (ii) introduce an integrated learning framework for routing. Our main contributions are: • We propose a causal end-to-end framework that learns routing policies by directly minimizing decision-making regret using observational data. Unlike the predominant decoupled paradigm, where various performance metrics (e.g., accuracy, cost) are first predicted and then used to inform routing decisions, our method integrates prediction and prescription into a unified objective. By optimizing for regret directly, the framework is explicitly aligned with the final routing decision quality. Furthermore, it is designed to scale efficiently and leverage readily available observational data, while accounting for treatment bias without requiring costly full-feedback datasets. • As the original regret minimization objective is not directly differentiable, we derive two surrogate objectives to enable end-to-end policy learning. The first is a classification-based upper bound that reframes regret minimization as a multiclass prediction problem under mild Lipschitz assumptions, allowing efficient training with standard methods. The second is a softmax-weighted regret surrogate that smoothly approximates regret using a softmax distribution and provably recovers optimal decisions at convergence. • We extend our framework to support heterogeneous cost preferences by introducing a unified model that conditions on both the query and the user’s cost sensitivity. Leveraging the affine structure of the utility function, we design an efficient interpolation scheme using only two endpoint models per interval. We theoretically show that the optimal treatment is piecewise constant in the cost parameter and that our architecture can exactly represent the optimal policy, enabling flexible and scalable routing across diverse preferences. • We conduct comprehensive experiments on two public benchmarks, demonstrating that our regret-minimizing and heterogeneous cost-aware approaches consistently outperform existing baselines. Our methods achieve state-of-the-art performance across both BERT-based and LLaMA-based embeddings, highlighting their robustness and practical effectiveness. 2 Methodology 2.1 Problem Formulation We consider a dataset of n observational samples, denoted by \mathcal{D}=\{(x_{i},t_{i},a_{i},c_{i})\}_{i=1}^{n} , where each sample is independently drawn from the joint distribution p(x,t,a,c) . Here, x_{i}\in\mathcal{X}\subset\mathbb{R}^{d} denotes a feature vector, typically an embedding, that characterizes the query; t_{i}\in[\mathcal{T}]:=\{0,1,\dots,T-1\} specifies the LLM assigned to the query; a_{i}\in\mathbb{R}_{\geq 0} denotes a numeric quality score of the LLM’s response, such as accuracy, or a preference rating; and c_{i}\in\mathbb{R}_{\geq 0} represents the cost incurred by model t_{i} when processing query x_{i} . Given a query x\in\mathcal{X} , the objective of an LLM router is to select a model t\in\mathcal{T} that maximizes the cost-aware performance, or utility, defined as y_{x}(t):=a_{x}(t)-\lambda\cdot c_{x}(t) . Here, \lambda\geq 0 is a user-specified parameter, modeling the trade-off between accuracy and cost. Higher values of y_{x}(t) , corresponding to greater performance, are preferred. 2.2 End-to-End Regret Minimization Our goal is to route each prompt x to the LLM t such that the decision leads to the highest possible utility y_{x}(t) . To this end, we aim to learn an end-to-end policy f:\mathcal{X}\to\mathcal{T} that minimizes the decision-making

Read the complete text