Efficient Evaluation of LLM Performance with Statistical Guarantees
Skyler Wu, Yash Nair, Emmanuel J. Candès · 2026
Read Efficient Evaluation of LLM Performance with Statistical Guarantees by Skyler Wu, Yash Nair, Emmanuel J. Candès (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20251.txt.
Introduction
Skyler Wu, Yash Nair, and Emmanuel J. Candès make the evaluation of large language models cheaper while keeping statistical guarantees. Testing many models on a large suite of benchmarks is expensive, since each test item costs a model call. The authors cast benchmarking as finite-population inference, like surveying a fixed set of items.
Under a fixed query budget, they seek tight confidence intervals for model accuracy with valid coverage. Smart sampling can then rank models without testing every item. This matters as benchmark suites grow: labs could spend less on evaluation and still know how sure each ranking is.
Catalog record
| CREATOR | Skyler Wu, Yash Nair, Emmanuel J. Candès |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20251
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Evaluation of LLM Performance with Statistical Guarantees Skyler Wu, Yash Nair, Emmanuel J. Candès Abstract Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with valid frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Proactive Active Inference —a finite-population extension of active inference ( 49 ) that enables direct question selection while preserving coverage. With negligible overhead cost, FAQ delivers up to 5 × 5\times effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels: this means that it matches the CI width of uniform sampling while using up to 5 × 5\times fewer queries. We release our source code and our curated datasets to support reproducible evaluation and future research. 1 Introduction Motivation Large language models (LLMs) are rapidly moving from research demos to deployed systems in domains such as healthcare, law, and education ( 7 ; 28 ; 11 ; 23 ; 3 ; 1 ) . As adoption grows, so does the number of candidate models organizations must evaluate and monitor. This scale is visible in public ecosystems: HuggingFace hosts at least 1.86 M models, including 300 K+ tagged as text-generation—many of them fine-tuned variants of certain base models ( 25 ) . Indeed, private institutions may maintain hundreds to thousands of internal variants, each exhibiting their own query-dependent strengths and weaknesses. As post-training becomes cheaper, the number of viable candidate models per deployment decision will only grow. A common practice is to evaluate candidate models’ performances on a finite, curated benchmark suite. Such suites increasingly include questions that (a) require expensive domain-expert adjudication (often by multiple annotators) ( 40 ; 26 ; 14 ; 10 ) ; (b) involve multi-turn interactions (e.g., helpfulness and safety) ( 21 ; 24 ) ; and/or (c) demand long-form reasoning with verification ( 4 ; 13 ) . With tens of thousands of questions (e.g., 12 K on MMLU-Pro ( 42 ) ), and hundreds to thousands of candidate models, exhaustively evaluating every model on every question is expensive in API costs, runtime, and expert annotation ( 47 ) . In practice, organizations may instead resort to “vibes”-based judgments on ad-hoc subsets ( 30 ) . Meanwhile, many organizations have access to historical performance data —from prior benchmark runs (e.g., the Open LLM Leaderboard ( 16 ) ) and/or production evaluation pipelines. As we show, this historical data can be leveraged to make model evaluation substantially faster and cheaper. Problem description Consider a benchmark suite of N_{q} questions, viewed as a fixed finite population. For a new model, let z_{j}\in\{0,1\} indicate whether it answers question j correctly. We wish to estimate the model’s accuracy over the entire finite question bank: \theta:=\frac{1}{N_{q}}\sum_{j=1}^{N_{q}}z_{j}\in[0,1]. (1) We also assume access to historical data H\in\{0,1,\text{NA}\}^{N_{\text{old}}\times N_{q}} from N_{\text{old}} prior models (e.g., previously benchmarked and/or observed in deployment), potentially with many missing entries (i.e., NA) due to resource and deployment constraints. Given a new model and budget n_{b} , the problem is: how should we adaptively select which n_{b} questions to query so as to construct a confidence interval (CI) for \theta that is as narrow as possible, while maintaining frequentist coverage? Here, coverage (say, 95\% ) means that over repeated runs of the evaluation procedure (i.e., over sampling randomness), the reported 95\% CI contains the true finite-bank accuracy \theta at least 95\% of the time. This guarantee is crucial for model selection and monitoring: miscalibrated uncertainty can lead to systematic overconfidence and poor deployment decisions, and the risk is amplified under adaptive evaluation where i.i.d. assumptions and heuristic error bars may fail. Such valid-coverage intervals provide uncertainty-quantified performance estimates to support deployment decisions alongside considerations such as cost and latency. Our contributions In this paper, we introduce Factorized Active Querying (FAQ), an efficient LLM evaluation method that can produce CIs of comparable width, using 5\times fewer queries and little to no additional computational costs, than competitive baselines (including uniform sampling), while maintaining frequentist coverage. FAQ has three components: (a) A Bayesian factor model that extracts information from partially-observed historical data. (b) A hybrid sampling policy combining variance reduction and active learning. (c) A coverage-preserving finite-population extension of active inference ( 49 ) , which we call Proactive Active Inference (PAI) . Unlike active inference, PAI directly selects the next question to query in a data-adaptive manner, rather than rigidly proceeding according to a fixed ordering and only making label/skip decisions. 1 1 1 While we focus on finite-bank benchmarking, PAI is a general coverage-preserving framework for adaptive querying over finite populations: we will explore its broader potential in future work. We evaluate FAQ on two benchmark suites under varying historical-data missingness and quantify the additional gains attributable to PAI. We release our compiled and curated datasets (with 4.4 K+ models and 21.5 K+ questions) to support reproducible benchmarking and future research. Datasets and code are available at https://github.com/skbwu/efficiently-evaluating-llms . 2 Related Work Probabilistic factor modeling FAQ is inspired by probabilistic factor modeling ( 31 ; 9 ; 6 ) , which assumes a (Bayesian) generative factor model for a partially-observed binary matrix—here, the matrix of model-question correctness outcomes—and learns low-dimensional latent factors from the data. Importantly, any uncertainty guarantees for these approaches are tied to correct specification of the factor model (and any prior assumptions). In contrast, FAQ uses the Bayesian factor model only as a mental model to guide adaptive query selection, while delivering valid frequentist coverage regardless of factor-model fidelity. If the data is well-described by the factor model, FAQ automatically adapts and delivers tighter CIs. Active statistical inference For model-free frequentist coverage, FAQ builds on active statistical inference ( 49 ; 18 ) , which uses machine learning predictions over unlabeled covariates/features to adaptively select labels under a fixed budget and construct CIs. Our benchmark setting, however, provides no explicit covariates, so we use the factor model above to construct latent features to guide adaptive query selection. Moreover, unlike sequential active inference—which, in a pre-specified “nature’s order” stream, can only make label/skip decisions for the next item and enforces the budget constraints only in expectation—PAI/FAQ directly selects the next question from the full benchmark bank using all available information and enforces the query budget exactly. Figure 1 : High-level overview of Factorized Active Querying (FAQ). (a) Using partially-observed historical outcomes H , we fit a historical model and question factors (Eq. 13 ). For a new model, we initialize its latent factor from the historical fit. Then, at each sampling round t , we (b) compute the hybrid sampling policy q_{t}(\cdot) from the current factor posterior (Eq. 10 ), (c) sample a question I_{t}\sim q_{t}(\cdot) , and (d) update the model factor using the observed outcome z_{I_{t}} (Eqs. 3 - 5 ). Finally, we (e) output the Proactive Active Inference (PAI) estimate and confidence interval (Eqs. 6 , 8 ), which guarantees model-free, frequentist coverage for finite-bank accuracy. Active learning FAQ draws on Bayesian