Importance Sampling is All You Need: Predict LLM’s performance on new benchmark by reusing existing benchmark. DOI: XXXXXXX.XXXXXXX CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Genera
Junjie Shi, Wei Ma, Shi Ying, Lingxiao Jiang, Yang Liu, Bo Du · 2025
Read Importance Sampling is All You Need: Predict LLM’s performance on new benchmark by reusing existing benchmark. DOI: XXXXXXX.XXXXXXX CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Genera by Junjie Shi, Wei Ma, Shi Ying, Lingxiao Jiang, Yang Liu, Bo Du (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01203.txt.
Introduction
Junjie Shi and co-authors proposed a shortcut in 2025 for guessing how a language model will score on a new coding test without running the test. Building good tests costs more and more, and test questions can leak into training data. Their method reads only the prompts of the new test.
The framework, BIS, reweights answers from old, graded tests using a classic sampling trick, with weights trimmed to keep guesses steady. Across eight thousand checks on four code models and nine tests, its guesses missed the true correctness score by about one percent on average. It also tracked pass rates well. The authors offer it as a cheap helper for building and vetting new tests.
Catalog record
| CREATOR | Junjie Shi, Wei Ma, Shi Ying, Lingxiao Jiang, Yang Liu, Bo Du |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01203
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Importance Sampling is All You Need: Predict LLM’s performance on new benchmark by reusing existing benchmark. DOI: XXXXXXX.XXXXXXX CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper Junjie Shi, Wei Ma, Shi Ying, Lingxiao Jiang, Yang Liu, Bo Du Abstract . With the rapid advancement of large language models (LLMs), code generation has become a key benchmark for evaluating LLM capabilities. However, existing benchmarks face two major challenges: (1) the escalating cost of constructing high-quality test suites and reference solutions, and (2) the increasing risk of data contamination, which undermines the reliability of benchmark-based evaluations. In this paper, we propose BIS , a prompt-centric evaluation framework that enables ground-truth-free prediction of LLM performance on code generation tasks. Rather than executing generated code, BIS estimates performance metrics by analyzing the prompt distribution alone. Built on importance sampling theory and implemented using Importance Weighted Autoencoders (IWAE), our method reweights samples from existing annotated benchmarks to estimate performance on new, unseen benchmarks. To stabilize the estimation, we introduce weight truncation strategies and compute marginal expectations across the fitted distributions. BIS serves as a complementary tool that supports benchmark development and validation under constrained resources, offering actionable and quick feedback for prompt selection and contamination assessment. We conduct extensive experiments involving 8,000 evaluation points across 4 CodeLlama models (7B–70B) and 9 diverse benchmarks. Our framework achieves an average absolute prediction error of 1.1% for code correctness scores, with best- and worst-case errors of 0.3% and 1.9%, respectively. It also generalizes well to other metrics, attaining average absolute errors of 2.15% for pass@1. These results demonstrate the reliability and broad applicability of BIS , which can significantly reduce the cost and effort of benchmarking LLMs in code-related tasks. 1. Introduction Large Language Models (LLMs) have demonstrated exceptional capabilities across various software engineering tasks, with code generation being a particularly notable example. As coding becomes one of the most critical benchmarks for assessing the performance of LLMs, it is essential to evaluate their coding abilities rigorously ( Chen et al. 2024 ) . To this end, numerous benchmarks have been developed to measure LLMs’ coding proficiency, including SWE-Bench ( Du et al. 2025 ) , HumanEval ( Chen et al. 2021 ) , and BigCodeBench ( Zhuo et al. 2024 ) . Despite their utility, current LLM evaluation benchmarks face two major challenges: High Benchmark Development Costs : Constructing reliable benchmarks requires significant manual effort, especially for developing detailed test suites and reference solutions. For example, BigCodeBench ( Zhuo et al. 2024 ) required the manual creation of over 5,000 tests. Attempts to automate test suite generation have been inadequate, with current automated methods achieving only around 39.2% accuracy ( Jain et al. 2024 ) . Risk of Data Contamination : Publicly available benchmarks and test suites can unintentionally become part of the LLM training datasets directly or indirectly, leading to artificially inflated performance over time and undermining benchmark reliability. Motivated by these limitations, we introduce a novel evaluation paradigm ( BIS , Prompt Importance Sampling) inspired by a previously overlooked insight: under fixed trained-well LLM parameters and evaluation criteria, prompt distributions inherently determine the LLM observed performance. By leveraging this relationship, we propose an evaluation framework that can estimate LLM performance by analyzing prompt distributions alone—without executing generated code or relying on costly ground-truth solutions. To the best of our knowledge, this is the first work to predict LLM code generation performance without ground-truth execution. Our framework is grounded in importance sampling, a statistical technique widely used in off-policy reinforcement learning ( Fujimoto et al. 2019 ; Munos et al. 2016 ; Uehara et al. 2022 ) , which allows us to estimate expectations under a target distribution using samples from a different, known distribution. In our setting, this means estimating the expected performance of an LLM on a new set of prompts by reweighting existing prompts from known benchmarks. We achieve this by modeling prompt distributions using Importance Weighted Autoencoders (IWAE), which are well-suited for capturing complex, multimodal distributions ( Burda et al. 2015 ) . By learning rich latent representations of prompt distributions, IWAE enables effective approximation of the importance weights, making it possible to reuse prior benchmark data to predict performance on novel code generation tasks. Our method complements ground-truth-based evaluation by alleviating two key challenges: reducing benchmark construction costs and mitigating data contamination risks. First, to mitigate high benchmark development costs, BIS enables early-stage performance estimation by transferring knowledge from existing, fully annotated benchmarks. This allows benchmark designers to assess the value of candidate tasks before committing to expensive test suite construction. Second, to help reduce the risk of data contamination, BIS provides an indirect signal of model familiarity with the prompt distribution. By analyzing discrepancies between expected and predicted performance patterns, it can help flag tasks that may overlap with training data, even in the absence of ground-truth execution. This complementary role positions BIS as a valuable tool for guiding benchmark design and validation, especially when operating under limited resources or concerns about data leakage. In theory, we analyze why BIS is effective and how it provides an unbiased estimation of LLM performance on the target benchmark. Furthermore, we discuss the practical feasibility of our approach. Additional details are provided in subsection 3.3 . We conducted extensive experiments involving approximately 8,000 data points across 4 LLMs (CodeLlama models ranging from 7B to 70B parameters) and 9 diverse benchmarks. Results demonstrate that our framework achieves an average prediction error as low as 0.9% on code correctness metrics, outperforming baseline distribution modeling methods. Additionally, our framework accurately generalizes to other crucial code-related metrics, including maintainability and security scores, validating its broad applicability. In summary, the contributions of this paper include: (1) Introducing the theoretical formalization explicitly linking prompt distributions to LLM performance, providing a rigorous conceptual foundation for prompt-based evaluation methods. (2) Proposing the first prompt-centric, ground-truth-free evaluation framework, BIS , tailored specifically for code-generation tasks, effectively addressing high benchmark development costs and data contamination risks. (3) Innovatively integrating importance sampling with IWAE, empirically validated through comprehensive experiments demonstrating improved accuracy, interpretability, and robustness. The remainder of the paper is organized as follows: section 2 discusses related work. section 3 presents our methodology. section 4 reports the experimental evaluation. section 5 discusses implications and limitations. section 6 concludes. 2. Related work 2.1. Code Benchmark So far, benchmarking for LLMs in code generation now encompasses a wide range of scenarios, including smart contracts ( Wijayakoon et al. 2025 ) , realistic environment settings ( Du et al. 2025 ) , and large-scale codebases ( Jimenez et al. 2023 ) . These benchmarks typically comprise at least two core components: (1) the code evaluation prompt, and (2) the test suite. Many benchmarks additionally incorporate reference solutions, problem sources, and other metadata. During evaluation, the LLM receives the code evaluation prompt as input and generates an output. This output undergoes formatting before being executed against the test suite. Consequently, a high-quality test