Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling

Mingqian Feng †, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Christopher White, Jianfeng Gao · 2026

Read Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling by Mingqian Feng †, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Christopher White, Jianfeng Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22636.txt.

Introduction

Mingqian Feng and a team measure the true attack risk of language models when foes get many tries. Safety tests often use one or a few attack prompts. Their 2026 preprint notes that real attackers can sample in parallel, again and again, until a harmful reply slips out.

Attack success grows with tries, so low-budget tests understate risk. The team build a statistical way to estimate risk under best-of-N sampling. Firm risk numbers help labs set honest safety bars. They also help buyers judge a model's true exposure. The work replaces a guess with a measure.

Catalog record

CREATORMingqian Feng †, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Christopher White, Jianfeng Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22636

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling Mingqian Feng †, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Christopher White, Jianfeng Gao Abstract Large Language Models (LLMs) are typically evaluated for safety under single-shot or low-budget adversarial prompting, which underestimates real-world risk. In practice, attackers can exploit large-scale parallel sampling to repeatedly probe a model until a harmful response is produced. While recent work shows that attack success increases with repeated sampling, principled methods for predicting large-scale adversarial risk remain limited. We propose a scaling-aware Best-of-N estimation of risk, SABER, for modeling jailbreak vulnerability under Best-of-N sampling. We model sample-level success probabilities using a Beta distribution, the conjugate prior of the Bernoulli distribution, and derive an analytic scaling law that enables reliable extrapolation of large-N attack success rates from small-budget measurements. Using only n=100 samples, our anchored estimator predicts ASR@1000 with a mean absolute error of 1.66, compared to 12.04 for the baseline, which is an 86.2% reduction in estimation error. Our results reveal heterogeneous risk scaling profiles and show that models appearing robust under standard evaluation can experience rapid nonlinear risk amplification under parallel adversarial pressure. This work provides a low-cost, scalable methodology for realistic LLM safety assessment. We will release our code and evaluation scripts upon publication to future research. 1 Introduction Large Language Models (LLMs) are increasingly deployed in real-world systems where safety failures can have serious societal and operational consequences. As a result, reliable evaluation of robustness under adversarial conditions has become a central research challenge. Most existing safety benchmarks assess robustness under single-shot or low-budget querying, implicitly assuming that adversaries have limited ability to repeatedly probe the model. However, in practice, attackers can automate large-scale parallel sampling pipelines, issuing thousands of attempts until a single harmful response is elicited, leading to systematic underestimation of real-world risk ( 2 ) . Recent work has begun to explore more realistic threat models for adversarial prompting. Particularly, Best-of-N Jailbreaking ( 9 ) demonstrates that attack success rates (ASR) increase sharply as the number of sampled prompt augmentations grows, even if variations are limited to simple character-level or surface-form manipulations, indicating that safety-aligned models can be compromised under sufficiently large attack budgets. While they convincingly establish the empirical existence of scaling vulnerabilities in such simplified settings, it mainly focuses on demonstrating the effectiveness of repeated sampling strategies and does not provide a principled statistical framework for estimating or extrapolating ASR to large budgets when exhaustive evaluation is computationally infeasible. Our key insight is that the scaling behavior of ASR is governed by the distribution of sample-level success probabilities, which we model as a Beta distribution, the conjugate prior of Bernoulli, providing a principled and analytically tractable representation of uncertainty in per-sample success rates. Under this modeling, we derive an analytic scaling law linking small-N observations to large-N attack success rates (ASR@N). Finally, we propose a scaling-aware Best-of-N estimation risk, SABER, bridging a critical gap between empirical attack evidence and predictive safety measurement. We evaluate our framework under a broader range of adversarial conditions that closely resemble real-world threat scenarios, including multiple advanced jailbreak pipelines operating in practical settings. This allows us to characterize not only the presence of scaling vulnerabilities but also their behavior under realistic operational constraints, offering a faithful view of LLM safety risk in deployment contexts. Our results show that large-scale adversarial risk can be accurately predicted using only small-budget measurements. For example, Using only budgets n=100 , SABER predicts ASR@1000 with a mean absolute error of 1.25 , compared to 12.04 for the naive baseline which is an 89.6\% error reduction. We further demonstrate that distinct attack methods exhibit heterogeneous Sample-ASR distributions, resulting in fundamentally different risk scaling profiles. These findings reveal that models which appear robust under standard evaluation (ASR@1) can experience rapid and nonlinear risk amplification under parallel adversarial pressure, underscoring the limitations of current evaluation practices and motivating the need for BoN-aware safety assessment. By introducing a mathematically grounded, low-cost, and scalable risk estimation framework, this work advances how jailbreak risk should be measured and reported. Rather than relying solely on expensive brute-force evaluation, we provide a principled methodology for forecasting adversarial risk, laying a foundation for next-generation LLM safety benchmarks that better reflect realistic adversarial settings. 2 Preliminaries Jailbreak attack and evaluation metric. Given a harmful query q , we model a jailbreak as a tripartite pipeline: an attacker \mathcal{A} produces an adversarial prompt q^{\text{adv}} ; the victim model \mathcal{V} generates a corresponding response r ; and a judge \mathcal{J} returns a score s and determines whether the victim \mathcal{V} is jailbroken. It can be formulated as follows: \begin{split}q^{\text{adv}}&\sim\pi_{\mathcal{A}}(\cdot|q),\\ r&\sim\pi_{\mathcal{V}}(\cdot|q^{\text{adv}}),\\ s&=\mathcal{J}(q,r)\in\{0,1\}\end{split} (1) where we define “ 1 ” as jailbroken and “ 0 ” as not jailbroken. When considering a dataset of harmful queries Q_{H}=\{q_{i}\}_{1}^{K} , Attack Success Rate (ASR) is a common metric to measure the overall adversarial risk. ASR is defined as the expected proportion of harmful queries in the dataset on which the attacker successfully jailbreaks the victim model. Scaling with Best-of-N sampling. By “ ASR ” without any suffix, we usually refer to the \text{ASR}@1 , which means the attacker only attempts once for each harmful query, \text{ASR}@1\coloneqq\mathbb{E}(\frac{1}{K}\sum_{i}^{K}s_{i}), (2) where s_{i} is the judge result on the i -th harmful query q_{i} . However, in practice, attackers can adopt a Best-of-N sampling strategy, repeatedly sampling adversarial prompts until one elicits a harmful response or the budget of N attempts is exhausted. Many prior works have demonstrated that this simple strategy effectively jailbreaks advanced open- and closed-source LLMs, using various attack methods, including simple text augmentation ( 9 ) . In this Best-of-N scenario, it is equivalent to consider N parallel attempts rather than sequential ones. This leads to the attack success rate at N attempts (ASR@N), formulated as below, \text{ASR}@N\coloneqq\mathbb{E}(\frac{1}{K}\sum_{i}^{K}s_{i,1}\lor...\lor s_{i,N}), (3) where s_{i,j}\in\{0,1\} is the judge result of the j -th independently sampled attempt on the i -th harmful query. Prior work has also shown that holding the adversarial prompt fixed while sampling multiple victim responses can yield successful jailbreaks. Moreover, since real-world LLM inference is inherently non-deterministic ( 8 ) , repeated sampling of the judge can exhibit analogous effects. Thus, to streamline the exposition without affecting the generality, we assume that each attempt independently samples the full attacker-victim-judge pipeline. Beta–Binomial distribution. As a result, a Best-of-N jailbreak attempt can be treated as a Bernoulli trial with an unknown success probability \theta . In that case, a common Bayesian choice is to place a Beta prior \theta\sim\mathrm{Beta}(\alpha,\beta) with \alpha,\beta>0 , because of its conjugacy to the Binomial likelihood. Specifically, the conjugacy implies that after observing k successes in n trials, the posterior remains Beta, \theta\mid k,n\sim\mathrm{Beta}(\alpha+k,\,\beta+n-k), (4) and the update depends only on the sufficient statistics (k,n-k) ( 5 ) . Integrating out \theta yields the corresponding Beta–Binomial marginal distribution for k , \Pr(k\mid n,\alpha,\beta)=\binom{n}{k}\frac{\mathrm{B}(k+\alpha,\,n-k+\beta)}{\mathrm{B}(\alpha,\beta)}. (5) 3 Scaling-Aware Best-of-N Estimation of Risk As shown in Figure 1 , for any given (attacker \mathcal{A} , victim \mathcal{V} , judge \mathcal{J} ), our SABER framework estimates \mathrm{ASR}@N from small-budget measurements in three stages. Initially, we collect the jailbreak outcomes of n attempts for each harmful query. Subsequently, we estimate the sample success probabilities to

Read the complete text