An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models

Hanrui Luo, Shreyank N Gowda · 2026

Read An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models by Hanrui Luo, Shreyank N Gowda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18775.txt.

Introduction

Luo and Gowda study multi-sample auditing for jailbreak detection in large language models, using the JailbreakBench Behaviors data set and several generator models of varying alignment. They test a lexical TF-IDF detector and an inconsistency detector across sampling budgets. Judging one output alone understates vulnerability, since more sampled outputs reveal more harmful behavior. Gains are largest moving from one output to moderate sampling, with larger budgets giving diminishing returns, and signals transfer partly across models. Code will be released.

Catalog record

CREATORHanrui Luo, Shreyank N Gowda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39912
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18775

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models Hanrui Luo, Shreyank N Gowda Abstract Detecting jailbreak behaviour in large language models remains challenging, particularly when strongly aligned models produce harmful outputs only rarely. In this work, we present an empirical study of output based jailbreak detection under realistic conditions using the JailbreakBench Behaviors dataset and multiple generator models with varying alignment strengths. We evaluate both a lexical TF-IDF detector and a generation inconsistency based detector across different sampling budgets. Our results show that single output evaluation systematically underestimates jailbreak vulnerability, as increasing the number of sampled generations reveals additional harmful behaviour. The most significant improvements occur when moving from a single generation to moderate sampling, while larger sampling budgets yield diminishing returns. Cross generator experiments demonstrate that detection signals partially generalise across models, with stronger transfer observed within related model families. A category level analysis further reveals that lexical detectors capture a mixture of behavioural signals and topic specific cues, rather than purely harmful behaviour. Overall, our findings suggest that moderate multi sample auditing provides a more reliable and practical approach for estimating model vulnerability and improving jailbreak detection in large language models. Code will be released. keywords Large Language Models, Jailbreak Detection, Multi-Generation Sampling, AI Safety, Adversarial Evaluation 1 Introduction Large language models (LLMs) are widely deployed across a range of applications, including search, content generation, coding assistance, and decision support systems IBM 2024 ; Pixelplex 2024 . Their rapid adoption has made them central to both consumer-facing products and high-stakes domains such as healthcare, finance, and legal services Shi et al. 2024 . This widespread use increases the importance of ensuring their reliability and safety. As these systems become more capable and accessible, concerns around robustness, controllability, and misuse have become central to research in trustworthy AI Yao et al. 2024 ; Dong et al. 2025 . Security is a critical aspect of LLM safety. Despite advances in alignment methods such as instruction tuning and reinforcement learning from human feedback (RLHF) Ouyang et al. 2022 , models remain vulnerable to adversarial prompting strategies, commonly referred to as jailbreak attacks Yi et al. 2024 ; Hakim et al. 2026 . These attacks exploit the interaction between instruction-following behaviour and safety constraints, leading models to generate harmful or policy-violating outputs even when safeguards are present Peng et al. 2024 . Prior work shows that such attacks are often easy to construct and effective across different model families and alignment settings Mustafa et al. 2025 ; Mustafa et al. 2026 . This suggests that current alignment techniques do not yet provide reliable safety guarantees. Most existing work has focused on improving alignment or designing detection methods. In contrast, the evaluation of jailbreak vulnerability itself has received less attention. A common limitation is that many studies rely on a single model output per prompt to assess both vulnerability and detection performance Mazeika et al. 2024 ; Zou et al. 2023 . While convenient, this approach assumes that one generation is representative of model behaviour. However, LLMs are stochastic and can produce different outputs for the same input Atil et al. 2024 ; Burnell et al. 2023 . In safety-critical settings, this variability is important. A model may refuse a harmful request in one instance and comply in another, which means single-output evaluation may underestimate the true vulnerability. Figure 1 motivates this and raises a key question: how much does single-output evaluation fail to capture the full range of model behaviour under adversarial prompting? Figure 1: Single-output evaluation underestimates jailbreak vulnerability. Multi-generation sampling reveals additional harmful behaviours, providing a closer estimate of true model risk. In this paper, we present an empirical study of output-based jailbreak detection under realistic conditions. We evaluate two detection approaches, a lexical baseline and a generation inconsistency-based method, across multiple generator models with varying alignment strength. We also analyse performance under different sampling budgets, from single-output evaluation to multi-generation settings, to understand how sampling affects both vulnerability estimation and detection reliability. Our results lead to three main findings. First, single-output evaluation underestimates jailbreak vulnerability, as additional harmful behaviour appears when more generations are sampled. Second, moderate multi-sample auditing, typically around three generations per prompt, captures most of the observable vulnerability, while larger sampling budgets provide limited additional benefit and can introduce variability. Third, detection signals partially generalise across models but are influenced by both behavioural patterns and lexical cues, as shown by cross-generator and category-level analyses. These findings suggest that multi-sample auditing provides a more reliable and practical framework for evaluating and detecting jailbreak behaviour in LLMs. Incorporating a small number of stochastic generations improves vulnerability estimation and detection effectiveness without significant computational cost. More broadly, this work highlights the need to move from single-output evaluation towards sampling-aware protocols that better reflect the stochastic nature of LLM behaviour in deployment. 2 Related Work 2.1 LLM Safety and Alignment Large language models are typically aligned using techniques such as instruction tuning and reinforcement learning from human feedback, which aim to reduce harmful or undesirable outputs by incorporating human preferences into the training process Ouyang et al. 2022 ; Bai et al. 2022 . These approaches encourage models to generate responses that are helpful, safe, and aligned with intended use. Over time, alignment strategies have significantly improved the average behaviour of language models across a range of tasks. However, a growing body of work has shown that alignment alone does not ensure robustness in adversarial settings. Even strongly aligned models can be induced to produce harmful or policy-violating outputs when exposed to carefully constructed inputs OpenAI 2023 . This suggests that alignment primarily addresses typical usage scenarios, while leaving models susceptible to targeted manipulation. As a result, security has become a central concern in the study of LLM safety, particularly in contexts where models are deployed in open or user-facing environments. 2.2 Jailbreak Attacks Jailbreak attacks represent a class of adversarial strategies designed to bypass alignment mechanisms. These attacks construct prompts that exploit the interaction between instruction-following behaviour and safety constraints, leading models to generate restricted or harmful content. Empirical studies have demonstrated that such attacks can achieve high success rates across a wide range of models, including those that have undergone extensive alignment procedures Shen et al. 2024 ; Zou et al. 2023 . An important property of jailbreak attacks is their ability to generalise across different model architectures. Prior work shows that adversarial prompts often transfer effectively between models, indicating that they exploit shared weaknesses rather than isolated implementation details Zou et al. 2023 . In addition, modern jailbreak techniques increasingly rely on indirect strategies such as prompt rewriting, role-playing, and multi-step reasoning to obscure malicious intent Yi et al. 2024 . Indirect prompt injection further extends this paradigm by embedding hidden instructions within external content that appears benign Greshake et al. 2023 . These developments make harmful intent more difficult to detect using straightforward surface-level analysis. 2.3 Jailbreak Detection Methods A range of approaches has been proposed to detect jailbreak attempts, which can be broadly categorised based on the type of information they utilise Hu et al. 2024 ; Xie et al. 2024 . Prompt-based methods focus on analysing the input prompt

Read the complete text