Expected Reward Prediction, with Applications to Model Routing

Kenan Hasanaliyev, Silas Alberti, Jenny Hamer, Dheeraj Rajagopal, Kevin Robinson, Jasper Snoek, Victor Veitch, Alexander Nicholas D’Amour · 2026

Read Expected Reward Prediction, with Applications to Model Routing by Kenan Hasanaliyev, Silas Alberti, Jenny Hamer, Dheeraj Rajagopal, Kevin Robinson, Jasper Snoek, Victor Veitch, Alexander Nicholas D’Amour (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20217.txt.

Introduction

Hasanaliyev and co-authors lift reward model scores from single responses to whole models, before any sampling. Reward models normally rank responses to one fixed prompt. The authors show the expected reward of a model on a prompt can be predicted directly. These predictions are sharp enough to route prompts between models at inference time. Routing aims to gain reward while holding compute cost down. On an open blended dataset, their simple routing beat category based baselines. The pool spanned Llama and Gemma models of several sizes.

Catalog record

CREATORKenan Hasanaliyev, Silas Alberti, Jenny Hamer, Dheeraj Rajagopal, Kevin Robinson, Jasper Snoek, Victor Veitch, Alexander Nicholas D’Amour
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35532
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20217

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Expected Reward Prediction, with Applications to Model Routing Kenan Hasanaliyev, Silas Alberti, Jenny Hamer, Dheeraj Rajagopal, Kevin Robinson, Jasper Snoek, Victor Veitch, Alexander Nicholas D’Amour Abstract Reward models are a standard tool to score responses from LLMs. Reward models are built to rank responses to a fixed prompt sampled from a single model, for example to choose the best of n n sampled responses. In this paper, we study whether scores from response-level reward models lifted to score a model’s suitability for a prompt, prior to seeing responses from that model. Specifically, we show that it is straightforward to predict the expected reward that an LLM would earn from the reward model under repeated sampling. Further, we show that these expected reward predictions are precise and discriminative enough to support an application to a model routing protocol that routes prompts to models at inference time to maximize reward while controlling computational cost. We demonstrate the performance of this routing procedure on the open-perfectblend dataset, using a model pool composed of Llama3.1-Instruct 8B/70B, Gemma2-IT 9B/27B, and Gemma1-IT 7B models. Our simple expected reward prediction–based routing (ERP) outperforms baselines that route prompts to models with the best average performance within each prompt’s category, and explains the success of more complex routing protocols that implicitly estimate an expected reward. Our approach has the added advantage of being trivially extensible as new models are added to the pool. 1 Introduction Reward models are commonly used in large language model (LLM) alignment, sampling, and evaluation [ 4 , 16 , 6 , 19 ] . A reward model is a function that takes a prompt x and a response y and returns a score r(x,y) quantifying how good the response is for the prompt. Notice that this makes no reference to language models—the reward is a property of text, not of a generative model. In this paper, we are interested in understanding how good a model is for a given prompt. More precisely, we are interested in lifting a reward function on responses to a reward function on models . The aim here can be understood as producing a LLM-level reward function that predicts a priori how well a random sample from the LLM can be expected to perform in response to the prompt x . Such a priori predictions would be useful for a number of inference time operations, such as model routing and prompt modification. Moreover, obtaining model-level rewards from response-level rewards would be especially convenient since there has already been enormous research and development effort put into creating high quality response-level rewards [ 9 ] . In this paper, we formalize the a priori reward of a language model \pi as \mdmathbb{E}_{Y\sim\pi(y\penalty\ |\penalty\ x)}[r(x,Y)] , the expected value of the response level reward of a random sample drawn from the LLM. The key question is whether it is possible to predict this quantity. It is not clear that predicting the expected reward a priori is actually possible, for at least two reasons. First, there is no guarantee that the generating LLM is well-behaved enough for its prompt-specific behavior to be easily predicted. There are two potentially unpredictable elements: (1) the sensitivity of the model to specifics of the prompt, and (2) the distribution of responses that a non-deterministic model can give to a prompt. If either aspect of the generating model is not well-behaved, it may not be practical to predict \mdmathbb{E}_{Y\sim P(y\penalty\ |\penalty\ x)}[r(x,Y)] , which summarizes a large range of behavior of the model. Secondly, the large majority of reward models used in practice are trained to encode the relative quality of two responses to the same prompt. That is, theoretically reward models only guarantee that r(x,y_{1})-r(x,y_{0}) is a meaningful quantity for all pairs of responses y_{0},y_{1} , but allow an arbitrary prompt-dependent value to be added to each reward (which cancels out when we take the difference). A model’s expected reward for a given prompt depends on this arbitrary value, and thus need not be learnable even in principle. 1 1 1 Formally, the expected value is not identified under the Bradley-Terry model. Thus, to the extent that prompt-wise expected reward is predictable, we would expect it to be a consequence of the implementation of the reward model as a fine-tuned pretrained language model. For these reasons, the scope of our contribution is primarily empirical. The main finding of this paper is that it is in fact possible to predict the expected reward with high fidelity using realistic language models and reward functions. Moreover, we find that this prediction is possible even using an astonishingly simple model: a linear probe trained on an off-the-shelf embedding representation of the prompt ( Figure 1(a) ). Finally, we demonstrate that this predictive power is not merely an artifact of identifying universally “good” or “bad” prompts: we show that predicted expected reward can be used to discriminate between models that would provide better or worse responses to the specific prompt. Specifically, we demonstrate this with an application to model routing. In the model routing application, we take in a prompt x and decide which model it should be optimally served to. We find that using lifted rewards leads to large improvements over any fixed model, particularly in applications where querying each model has variable cost ( Figure 1(b) ). 1.0 0.2 0.0 0.4 (3.2,-1.1,0.7) (a) (b) Figure 1 : (Left) Expected reward prediction workflow. We train a linear model on an embedding of the prompt to predict the expected reward that the model would earn responding to the prompt. (Right) Expected reward prediction (ERP) can be used to route queries to the best, cost-effective model for that query. The Pareto frontier (blue) shows that an ERP-based routing policy dominates baselines that don’t discriminate between models in the context of each prompt, regardless of cost sensitivity. Additional regret-cost tradeoff plots for other RMs can be found in Figure 6 . 2 Preliminaries 2.1 Language Models In this paper, we view a large language model \pi as a kernel that maps prompts x to probability distributions \pi(Y\penalty\ |\penalty\ x) over responses y . Our interest is in predicting aspects of the response a priori . Formally, this is equivalent to characterizing (properties of) the probability distribution \pi(Y\penalty\ |\penalty\ x) from the prompt. In particular, we will be interested in learning functions of the form \text{ER}_{\pi}(x):=\mdmathbb{E}_{Y\sim\pi(Y\penalty\ |\penalty\ x)}[r(x,Y)] . 2.2 Reward Modeling A central problem in generative language model evaluation is scoring open-ended outputs that vary on a large number of dimensions that are meaningful to humans. Pre-specifying these dimensions and scoring responses along each would be onerous, especially for hard-to-define concepts like “helpfulness”. Reward models avoid this problem by inferring an overall notion of “goodness” that is revealed by preference data. Preference data has the form (x,y_{-},y_{+}) , where x is the input prompt, y_{+} and y_{-} are the preferred and dispreferred candidate responses. A reward model translates this preference-level data into a response-wise score r(x,y) , so that for a given prompt, r(x,y)>r(x,y^{\prime}) implies that y is more likely to be preferred to y^{\prime} by raters that generated the preference data. The standard approach to reward modeling is to do this translation by maximizing

Read the complete text