Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling

Indranil Halder, Cengiz Pehlevan · 2025

Read Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling by Indranil Halder, Cengiz Pehlevan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19905.txt.

Introduction

Indranil Halder and Cengiz Pehlevan build a solvable toy model of a fashionable practice: letting a language model judge its own multiple attempts. Inference-time scaling spends compute at answer time by sampling many candidates and selecting with a reward, here supplied by a judge model. The principles behind when that helps remain murky, so the authors study Bayesian linear regression with a reward-weighted sampler as an analytically tractable stand-in.

In the high-dimensional regime they derive closed forms for the predictive mean and variance, then track generalization error as samples accumulate. With a well-matched reward, error falls monotonically, and in the best-of-k teacher-reward limit it decays quadratically in the number of samples. A badly misspecified reward instead creates a finite optimal sample count beyond which more sampling hurts, and task difficulty erodes the whole advantage.

Catalog record

CREATORIndranil Halder, Cengiz Pehlevan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19905

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling Indranil Halder, Cengiz Pehlevan Abstract Recent developments in large language models have shown advantages in reallocating a notable share of computational resource from training time to inference time. However, the principles behind inference time scaling are not well understood. In this paper, we introduce an analytically tractable model of inference-time scaling: Bayesian linear regression with a reward-weighted sampler, where the reward is determined from a linear model, modeling LLM-as-a-judge scenario. We study this problem in the high-dimensional regime, where the deterministic equivalents dictate a closed-form expression for the posterior predictive mean and variance. We analyze the generalization error when training data are sampled from a teacher model. We draw k k inference-time samples and select via softmax at a temperature applied to a quadratic reward. When the reward is not too different from the teacher, the generalization error decreases monotonically with increasing inference time samples k k . However, the specific reward that optimizes inference-time selection generally differs from the teacher. In contrast, substantial reward misspecification induces a finite optimal k k beyond which more sampling can increase the generalization error. For fixed k k , there exists an optimal sampling temperature. We experimentally verify these facts in large language model inference with an additional large language model as a judge. In the “best-of- k k ” limit with the teacher as reward, we theoretically show that the generalization error decays as Θ ⁡ ( 1 / k 2 ) \Theta(1/k^{2}) and determine the leading coefficient via extreme value theory. These formulas delineate domains where scaling inference-time computation is provably preferable to collecting more data. Finally, we demonstrate that when task difficulty increases, the previously mentioned advantage of inference-time compute degrades. Code: GitHub repository 1 Introduction Scaling training compute via larger models and more data drives dramatic gains ( Hestness et al. 2017 ; Kaplan et al. 2020 ; Hoffmann et al. 2022 ) . In parallel, across tasks, allowing models to ‘think longer’ at inference by sampling multiple candidates, re-ranking with a reward, or aggregating votes consistently improves precision and reliability ( Wang et al. 2023 ; Zheng et al. 2023 ; Zhang et al. 2024 ; Wu et al. 2024 ; Snell et al. 2025 ) . Best-of- k (choose the highest-reward sample based on the feedback of a judge LLM) and majority voting (choose the consensus among the generated answers) have become standard inference-time tools ( Brown et al. 2024 ; Schaeffer et al. 2025a ; Chen et al. 2024a ; Huang et al. 2025a ) along with self-verification ( Saunders et al. 2022 ; Weng et al. 2023 ) . Despite widespread adoption, key questions for inference-time computation lack crisp answers. How to optimally configure inference-time sampling to minimize generalization error under realistic compute constraints and how to allocate compute optimally between pretraining and inference time remain open questions ( Wu et al. 2024 ) . Which reward model should we use for inference? What is the appropriate inference-time sampler, e. g., temperature settings? How large should k be and when do more samples stop helping? How should we allocate a fixed compute budget between training and inference to minimize generalization error? We lack a simple, solvable model that gives us intuitions and actionable prescriptions. To fill this gap, we propose a minimal and analytically tractable setting: Bayesian regression based on a teacher-student setting with a controlled reward and a temperature-dependent inference-time sampler, in which best-of- k appears as limit. This setup is a proxy for LLM-as-a-judge providing reward for the generated answers from a base-LLM. We theoretically study the generalization error \delta as a function of the size of the training data set n , the dimension of the data d , the number of inference time samples k , the sampling temperature T and the reward parameter \mathbf{w}_{R} and demonstrate various optimality conditions on these parameters. This simple model recapitulates existing findings on large language model inference and yields novel predictions that we evaluate in this paper. It reproduces the empirical observation that unbounded increases in the number of inference-time samples do not confer additional benefit ( Snell et al. 2025 ) . Furthermore, it predicts the existence of an optimal temperature for the reward process. Further, strong rewards shift optimal temperature to a lower value. We empirically examine and confirm this prediction using Meta-Llama-3-8B-Instruct with Mistral-7B-Instruct-v0.3 serving as the judging model. (a) T=20\sigma^{2} (b) T=10\sigma^{2} Figure 1: In the plot the radial distance is the magnitude c of the vector \mathbf{w}_{R}-\mathbf{w}_{T} and the polar variable is the angle \theta between \mathbf{w}_{R}-\mathbf{w}_{T} and \mathbf{w}_{T} . We have chosen S=1,\sigma=10^{-4},\gamma=10^{-3},d=2,n=10^{4} and sampled teacher weight \mathbf{w}_{T}=(c\cos\theta_{T},c\sin\theta_{T})\sim\mathcal{N}(0,2^{2}\mathbf{I}) . We have parameterized the reward weight as follows: \mathbf{w}_{R}=\mathbf{w}_{T}+(c\cos(\theta_{T}+\theta),c\sin(\theta_{T}+\theta)),\theta\in[0,2\pi) . See section 2 for details of notation and conventions. From the plot, we see that as temperature T decreases the domain, where generalization error \delta decreases monotonically with the increase in inference-time samples k , shrinks. Similar result is presented in remark 2 for the proportional limit. At the technical level our contributions are the following: • We propose a solvable model for inference-time scaling: Bayesian regression where the ground truth is given by the teacher model y=\mathbf{w}_{T}\cdot\mathbf{x}/\sqrt{d} and the reward function is quadratic r(y,\mathbf{x})=-(y-\mathbf{w}_{R}\cdot\mathbf{x}/\sqrt{d})^{2} . We generate k samples at inference and choose using a softmax at temperature T over the reward. This method is conceptually close to the importance sampling method in Faria and Smith 2025 . We present a formula for the generalization error \delta in the proportional limit: d\to\infty,n\to\infty with \alpha=d/n fixed. • In order to derive analytic insights about the inference time optimization, we derive a series expansion for \delta around large T , making explicit its dependence on n , k , and the alignment between \mathbf{w}_{R} and \mathbf{w}_{T} . The analytical expansion shows a sharp dependence on reward quality: when \mathbf{w}_{R} is sufficiently close to \mathbf{w}_{T} , i.e., small \|\mathbf{w}_{R}-\mathbf{w}_{T}\|/\|\mathbf{w}_{T}\| , increasing k monotonically decreases \delta , and the reward \mathbf{w}_{R} that optimizes inference-time selection generally differs from the data-generating teacher \mathbf{w}_{T} . In contrast, when \mathbf{w}_{R} is poorly aligned, \delta is non-monotone in k , yielding an optimal finite k (see Figure 1 ), echoing phenomena observed empirically in large language models ( Snell et al. 2025 ) . Furthermore we show that at fixed k , there exists an optimal temperature T for the rewarding process. Similar yet distinct observation has been made in large language models ( Du et al. 2025 ) for sampling temperature of the model itself. We emphasize that our result is about the sampling temperature of the rewarding process - different from the sampling temperature of the language model. • To explore the best-of-k case, we consider the T=0 limit. Using extreme value theory, we analytically prove that the expectation value of \delta scales as \Theta(1/k^{2}) at large k when we have access to the teacher, i.e, \mathbf{w}_{R}=\mathbf{w}_{T} . Based on our theoretical analysis in best-of- k limit, we quantify the parametric region where scaling inference-time compute is more beneficial compared to training compute. Finally we note that when task difficulty increases, the previously mentioned advantage of inference-time compute degrades. • To test our predictions in realistic

Read the complete text