Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization

Elad Tolochinsky Yaniv Tenzer Yaniv Romano · 2026

Read Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization by Elad Tolochinsky Yaniv Tenzer Yaniv Romano (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10405.txt.

Introduction

Elad Tolochinsky, Yaniv Tenzer, and Yaniv Romano cut the cost of picking the best language model for a benchmark. Exhaustive evaluation, running every model on every example, is expensive: one GAIA evaluation can cost up to 2,829 dollars. Multi-armed bandit algorithms reduce calls by sequentially choosing the next model-example pair, and low-rank factorization can predict scores from a partially observed score matrix. Their 2026 paper combines those savings without sacrificing validity.

Predicted scores are not ground truth: bias can creep in, and the best model can be misidentified. The authors build doubly robust estimators for each model's performance. Those estimators use low-rank predictions to reduce variance, enabling valid finite-sample confidence ranges even with adaptive model choice and sampling without replacement. On real-world benchmarks the approach reduced required evaluations, saving compute and cost while picking out the best-performing model correctly.

Catalog record

CREATORElad Tolochinsky Yaniv Tenzer Yaniv Romano
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10405

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization Elad Tolochinsky Yaniv Tenzer Yaniv Romano Abstract Selecting the best large language model (LLM) for a fixed benchmark is often expensive, since exhaustive evaluation requires running every model on every example. Multi-armed bandit (MAB) algorithms can reduce the number of LLM calls by sequentially selecting the next model–example pair to evaluate, thereby avoiding wasted evaluations on clearly underperforming models. Further savings can be achieved by predicting model scores from the partially observed model–example score matrix using low-rank factorization. However, such predictions are not ground truth: they can be biased and may therefore lead to incorrect identification of the best model. In this work, we propose a principled framework that combines MAB with cheap predicted scores without compromising statistical validity. Specifically, we derive doubly robust estimators of each model’s performance that use the low-rank predictions to reduce variance. This enables the construction of valid finite-sample confidence intervals in our setting, where models are selected adaptively and examples are sampled without replacement. Empirical results on real-world benchmarks show that our approach reduces the number of required evaluations, yielding meaningful savings in compute and cost while accurately identifying the best-performing model. 1 Introduction LLMs are employed across a wide range of applications, including customer-facing dialogue systems, code generation, writing assistance, and information retrieval [ 1 , 2 , 3 ] . As LLM adoption grows, practitioners often encounter the challenge of choosing the best-performing model and configuration—such as prompts and hyperparameters—for a specific task. A common practice for that purpose is to compare the performances of the models on fixed benchmarks using a score function. For example, consider the task of mathematical problem solving, as studied in benchmarks such as MMLU [ 4 ] and MATH [ 5 ] . In this case, a score function can be a binary function that indicates whether the model’s solution is correct. Unfortunately, exhaustively evaluating every model on every example is resource-intensive: a single evaluation on the GAIA benchmark [ 6 ] can cost up to $2,829 [ 7 ] , and the recent Holistic Agent Leaderboard evaluation spent roughly $40,000 to compare 9 models across 9 benchmarks [ 8 ] . Moreover, these are single-run figures: stochastic agents require repeated rollouts to produce reliable estimates, multiplying costs further. To reduce the evaluation cost, recent works utilize multi-armed bandit (MAB) algorithms, such as UCB-E [ 9 ] , to adaptively allocate the limited evaluation budget. Ideally, MAB should spend more budget on promising candidates and less on clearly underperforming ones. In this framework, each model is an arm, and pulling an arm corresponds to evaluating the model on examples and observing the scores. The evaluation proceeds sequentially: at each step, the algorithm selects a model and a batch of examples that this model has not yet evaluated, computes their scores, and updates its estimate of the model’s mean accuracy. These updated estimates are then used to guide the selection of the next model. Further computational savings can be achieved by using cheap predicted scores rather than the exact ones. Such predictions can be found in the structure of the model-example score matrix, i.e., the matrix whose (i,j) entry is the score obtained by running the i -th model on the j -th example. Concretely, the work in [ 10 ] utilizes the fact that models may exhibit similar performance profiles across examples, and examples may admit similar scores across models. This correlation suggests that the full score matrix may be well approximated by a low-rank matrix. During evaluation, this matrix is only partially observed. Yet, low-rank factorization provides a cheap way to impute the missing entries, producing predicted scores that can guide MAB’s evaluation procedure. Indeed, it is shown in [ 10 ] that such predictions can accelerate the identification of the best model. The challenge is that these predicted scores are not ground truth. They can be biased and may therefore lead to incorrect identification of the best model. This raises the need for a principled framework that incorporates predicted scores while maintaining unbiased estimates of the models’ mean performance. To this end, we build on doubly robust estimators—including augmented inverse probability weighting (AIPW) [ 11 , 12 ] and prediction-powered inference (PPI) [ 13 , 14 ] —that enable the construction of unbiased estimators of population parameters with potentially lower variance by leveraging machine learning (ML) predictions. These tools have been developed for adaptive MAB, contextual bandits, and even best-arm identification [ 15 , 16 , 17 , 18 ] . However, it is unclear how to extend the existing guarantees for these techniques to the finite-population setting considered in this paper. Beyond sampling examples without replacement from a fixed benchmark, our setting introduces an additional complication: the imperfect predictions are obtained by repeatedly refitting an online low-rank model to the same adaptively observed model–example score matrix, resulting in strong cross-arm dependence. In this work, we bridge the gap by presenting PULSE: Prediction-Powered Unbiased Low-Rank Sequential Evaluation. Given a fixed budget of LLM calls, PULSE leverages imperfect predicted scores to increase the probability of identifying the best-performing model. Importantly, our approach remains statistically valid even when the predictions are inaccurate. As we show, when predictions are informative, our method accelerates identification; when they are not, it can approach standard MAB behavior that relies solely on observed evaluations. Our main contributions are as follows: • Our first key contribution (Section 3 ) is an estimator of each model’s mean accuracy that leverages predictions obtained by low-rank factorization of the score matrix. The resulting estimator is guaranteed to be unbiased, while informative predictions can reduce its variance. We also show how to construct confidence intervals (CIs) for the models’ mean accuracies and establish their finite-sample validity. • Building on our theoretical results, our second key contribution (Section 3 ) is a MAB algorithm for best-model identification. We establish a finite-sample success guarantee for PULSE and analyze how prediction quality can affect its convergence. • Finally, we evaluate the performance of our algorithm on 2.2K models and six real-world benchmarks. As shown in Figure 1 , our experiments demonstrate that PULSE can save up to 46\% of LLM calls relative to UCB-E when targeting 95\% identification accuracy. Code: https://github.com/elad-tolo/pulse-llm-eval . Figure 1 : Budget needed for 95% best-model identification accuracy (lower is better). Bench 1 and 2 cover 2.2K models each. \Delta_{\text{top2}} , the gap between the top two models, controls difficulty. See Section 4 . 2 Problem Formulation and Related Work We let \mathcal{X}=\{x_{1},\ldots,x_{n}\} be a fixed dataset of examples, \mathcal{M}=\{f_{1},\ldots,f_{m}\} be a set of LLMs. Let s(M(x))\in\{0,1\} be a scoring function used to score the output of the LLMs. Without loss of generality, we assume that s(f(x))>s(f^{\prime}(x)) indicates that model f has better performance than model f^{\prime} on example x . We denote by S\in\{0,1\}^{m\times n} the score matrix so that S_{ij}:=s(f_{i}(x_{j})) . In our setting, S is only partially observed, and we think of its entries as random variables. More precisely, let S_{i} be the score obtained by model i when tested on an example that is drawn uniformly from \mathcal{X} . Under this formulation, \mu_{i}=\mathbb{E}[S_{i}]=\frac{1}{n}\sum_{j=1}^{n}S_{i,j} , where the expectation is taken with respect to

Read the complete text