QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks
Taylor Lundy, Narun K. Raman, & Kevin Leyton-Brown · 2026
Read QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks by Taylor Lundy, Narun K. Raman, & Kevin Leyton-Brown (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17842.txt.
Introduction
Lundy and two colleagues tackle dynamic language-model benchmarks. Templates there generate endless question variants, and finding weak spots gets costly. They adapt COUP, a Bayesian optimization method, for practical model pipelines. They wrap it in a tool letting users target question types they care about, such as unusually hard items relative to measured complexity. Their method, QuickScope, finds genuinely hard questions with fewer samples than standard baselines while cutting false alarms from noisy results.
Catalog record
| CREATOR | Taylor Lundy, Narun K. Raman, & Kevin Leyton-Brown |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60001 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17842
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks Taylor Lundy, Narun K. Raman, & Kevin Leyton-Brown Abstract LLM benchmarks are increasingly dynamic: instead of containing a fixed set of questions, they define templates and parameters that can generate an effectively unlimited number of question variants. This flexibility is valuable, but it makes evaluation expensive—especially when the goal is not just determining an average score, but reliably identifying a model’s weak spots. This paper introduces a new methodology for identifying hard questions in dynamic benchmarks. It leverages COUP, a recent Bayesian optimization algorithm (Graham, Velez & Leyton-Brown, 2026), after introducing several substantive modifications to make the algorithm suitable for practical LLM pipelines. We also wrap it in a tool that supports flexible choices of datasets and utility functions, enabling users to target the kinds of questions they care about (e.g., low-accuracy questions; questions that are unusually hard relative to their measured complexity). In experiments across a range of benchmarks, we show that our method, dubbed QuickScope , discovers truly difficult questions more sample efficiently than standard baselines, while also reducing false positives from noisy outcomes. 1 Introduction Benchmarking is one of the main ways we track progress in large language models (LLMs). But as model performance rises, two practical issues have become increasingly hard to ignore. First, static test sets are growing increasingly vulnerable to overfitting: repeated optimization against a fixed benchmark can encourage improvements that reflect adaptation to the benchmark rather than underlying capability. Second, at the frontier, models are strong enough that overall accuracy differences can be small, and what we need from benchmarks is often diagnostic: to support downstream decisions, it matters which problems a model misses and whether those failures persist when we restate the question in slightly different words or resample the names and numbers without changing the underlying problem. However, such fine-grained diagnostics are difficult to obtain from static benchmarks, which typically contain only a small number of questions per concept and often lack a clear notion of similarity between questions. In response, there is a growing tendency to make benchmarks dynamic. Instead of a fixed list of questions, a dynamic benchmark specifies parameters that can generate an effectively unbounded set of questions. For example, a question might be parameterized by the underlying concept it evaluates (e.g., addition), its domain (e.g., a medical setting) and its perspective (e.g., first person) and then the question can be instantiated using randomly generated values. This makes it possible for many i.i.d. questions to be instantiated, improving the statistical power with which performance on a concept can be measured. Dynamic benchmarks most significantly eclipse static alternatives when the goal is to identify a model’s weak spots, as compared e.g. to its average performance under some sampling distribution. In such cases, it is important to surface questions that are truly difficult for a target model, rather than simply highlighting occasional failures due to stochasticity, formatting quirks, or other transient effects. Put differently, the goal is not merely to find failures, but to find questions or regions that can be certified as hard with high confidence. Such certification requires prospective weak regions to be sampled densely. It is thus necessary either to employ uniform random sampling and to sample easy regions just as densely, or to employ some dynamic sampling strategy. This paper takes the dynamic approach, framing evaluation on dynamic benchmarks as a sequential decision problem. Each query incurs cost; outcomes are noisy; and the space of possible questions is structured by the benchmark’s parameterization. The evaluator’s job is to allocate budget adaptively to match an explicit user objective. The objective might be “find low-accuracy questions,” but it can also be richer, such as “find questions that are unusually hard relative to their measured complexity” or “find small changes in the question that cause a big drop in accuracy.” Our approach brings Bayesian optimization to benchmark evaluation by building on COUP ( 12 ; 11 ) , a recent UCB algorithm originally developed for cost-aware algorithm configuration. We treat each benchmark template as a configuration and use the user-defined objective to define payoffs; COUP then decides which parts of the benchmark to query next by sampling based on uncertainty-aware upper confidence bounds. However, using COUP off-the-shelf is not sufficient for modern LLM pipelines. LLM evaluation is naturally batched and parallel, and many preferences are over aggregate performance (e.g., long-run accuracy) rather than per-sample performance. We therefore modify COUP to (i) operate effectively under parallel evaluation and (ii) support utility mappings that let users express preferences over long-run aggregates instead of single outcomes. We package these ideas into a practical tool for dynamic benchmarks. We call this tool QuickScope —it quickly scopes out a model’s weak spots within a benchmark’s configuration space—and it cleanly separates (a) the benchmark interface that maps a configuration to a concrete question instance, (b) the model interface that executes the evaluation, and (c) a utility function that captures the user’s notion of “interesting” or “hard.” This separation is intentional: dynamic benchmarks vary widely in how their parameters are structured and how questions are instantiated, and different evaluation goals call for different utility definitions. By making these components explicit and swappable, the same underlying algorithm can target low-accuracy regions, prioritize questions that are hard conditional on complexity, or implement other user-specified objectives without requiring any changes to the core optimization logic. Empirically, across benchmarks, we show that our method discovers truly difficult questions with more sample efficiency than standard baselines, while reducing false positives that arise from noisy outcomes. In settings where users care about reliable identification of weak spots, this translates into fewer total model calls to obtain a comparable (or better) set of certified-hard items. The remainder of this paper proceeds as follows. Section 2 reviews prior work on LLM evaluation, dynamic benchmarks, and algorithm configuration methods; it also provides the COUP background needed for our setting, including its anytime guarantees and the roles of its confidence bounds and proposal mechanism. With this setup in place, Section 3 introduces our extensions that make COUP applicable to practical LLM pipelines—in particular, batched/parallel evaluation and utility functions defined over long-run aggregates rather than single noisy outcomes. Section 4 describes our tool, QuickScope , and its extension points. Section 5 evaluates our approach against standard baselines across benchmarks, and Section 6 discusses limitations and broader implications for evaluation design. 2 Related Work and Background LLM Evaluation A large literature studies how to make LLM evaluation more informative than a single headline score, including broader metric suites, robustness checks, and controlled prompting protocols ( 21 ; 7 ; 26 ; 1 ; 3 ; 31 , e.g.,) . Another growing body of work highlights the fragility of static benchmarks under repeated iteration, training–test overlap, and shifting evaluation conditions ( 18 ; 30 ; 10 ; 5 ; 6 ; 8 ; 4 ; 29 , e.g.,) . These concerns motivate dynamic benchmarks, where a benchmark specifies generators or templates that can produce many question variants rather than a fixed test set. Recent work has begun to explore how to search such spaces efficiently. Most of these benchmarks generate questions from formal rules, simulators, or solvers ( 17 ; 24