Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models

Yakai Li 1,2 , Jiekang Hu 1,2 , Weiduan Sang 3 , Luping Ma 1 , Dongsheng Nie 4, Weijuan Zhang 1,2 , Aimin Yu 1,2 , Yi Su 4 , Qingjia Huang 1,2 , Qihang Zhou 2 · 2025

Read Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models by Yakai Li 1,2 , Jiekang Hu 1,2 , Weiduan Sang 3 , Luping Ma 1 , Dongsheng Nie 4, Weijuan Zhang 1,2 , Aimin Yu 1,2 , Yi Su 4 , Qingjia Huang 1,2 , Qihang Zhou 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21038.txt.

Introduction

Yakai Li and co-authors expose a blind spot in language model safety: attacks that prefill the start of the model's reply. Their 2025 paper studies a feature that lets users dictate how a reply begins. That control shifts attacks from persuading the model to seizing its state. The team sorts the new attack kinds and tests them as a black box, with no view inside the models. Fourteen models face the tests.

Success rates run high, and adaptive forms pass nearly every guard on several models. Token-level checks show why. The prefilled start flips the first token from refusal toward compliance, and the reply follows its own lead. Prefills also boost older prompt attacks by a clear margin. Standard content filters block little. A detector that watches the tie between prompt and prefill works better. The authors urge safety training to cover this surface.

Catalog record

CREATORYakai Li 1,2 , Jiekang Hu 1,2 , Weiduan Sang 3 , Luping Ma 1 , Dongsheng Nie 4, Weijuan Zhang 1,2 , Aimin Yu 1,2 , Yi Su 4 , Qingjia Huang 1,2 , Qihang Zhou 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21038

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models Yakai Li 1,2 , Jiekang Hu 1,2 , Weiduan Sang 3 , Luping Ma 1 , Dongsheng Nie 4, Weijuan Zhang 1,2 , Aimin Yu 1,2 , Yi Su 4 , Qingjia Huang 1,2 , Qihang Zhou 2 Abstract Warning: this paper includes examples that may be offensive or harmful. Large Language Models face security threats from jailbreak attacks. Existing research has predominantly focused on prompt-level attacks while largely ignoring the underexplored attack surface of user-controlled response prefilling. This functionality allows an attacker to dictate the beginning of a model’s output, thereby shifting the attack paradigm from persuasion to direct state manipulation. In this paper, we present a systematic black-box security analysis of prefill-level jailbreak attacks. We categorize these new attacks and evaluate their effectiveness across fourteen language models. Our experiments show that prefill-level attacks achieve high success rates, with adaptive methods exceeding 99% on several models. Token-level probability analysis reveals that these attacks work through initial-state manipulation by changing the first-token probability from refusal to compliance. Furthermore, we show that prefill-level jailbreak can act as effective enhancers, increasing the success of existing prompt-level attacks by 10 to 15 percentage points. Our evaluation of several defense strategies indicates that conventional content filters offer limited protection. We find that a detection method focusing on the manipulative relationship between the prompt and the prefill is more effective. Our findings reveal a gap in current LLM safety alignment and highlight the need to address the prefill attack surface in future safety training. I Introduction Large Language Models (LLMs), including ChatGPT [ 1 ] , DeepSeek [ 2 ] , and Claude [ 3 ] , are increasingly deployed in diverse applications, from conversational assistants and code optimization tools to content generation and data augmentation systems [ 4 , 5 ] . However, these models face persistent security challenges from adversarial attacks that attempt to elicit harmful content through carefully crafted inputs [ 6 ] . These attacks, commonly known as jailbreak attacks [ 7 ] , are typically categorized into two main approaches: white-box attacks that require access to model internals for gradient-based optimization (e.g., GCG [ 8 ] , AutoDAN [ 9 ] ), and black-box attacks that operate solely through the model’s input-output interface, including prompt engineering techniques such as DAN [ 10 ] and iterative optimization methods like PAIR [ 11 ] and ReNeLLM [ 12 ] . Fig. 1: Comparison of prompt-level versus prefill-level jailbreak attack. Existing jailbreak research has primarily focused on manipulating user prompts to circumvent safety mechanisms. These prompt-level approaches employ various strategies, including role-playing attacks where models are instructed to assume personas that disregard safety constraints [ 13 , 14 ] , automated prompt optimization methods that iteratively refine adversarial inputs [ 11 , 12 ] , and encoding techniques that obscure malicious intent through character manipulation or linguistic obfuscation [ 15 , 16 ] . Multi-turn conversation attacks exploit dialogue context to gradually build toward harmful outputs [ 17 ] , while persona modulation techniques leverage character-based manipulation to bypass safety filters [ 18 ] . Despite achieving notable success rates across various models, these methods fundamentally operate within the constraints of prompt-level manipulation, requiring the attacker to persuade the model to generate prohibited content through input crafting alone. However, these prompt-level approaches overlook an underexplored attack surface available at the API layer: user-controlled response prefilling. This functionality, illustrated in Figure 1 , allows users to specify the initial portion of the model’s response, effectively bypassing the decision-making process that typically triggers safety refusals. Unlike prompt-level attacks that must convince the model to generate harmful content, prefill-level attacks can directly force compliance by manipulating the initial generation state. This represents a shift from persuasion-based to state-manipulation-based adversarial strategies. Prefill functionality is supported across multiple commercial providers [ 19 , 20 , 21 ] and is widely available in open-source deployments. While originally designed for legitimate use cases such as structured output formatting and conversation continuation, its security implications remain largely unexplored, creating a potential blind spot in current safety measures. To address this gap, we present the first systematic security analysis of prefill-level jailbreak attacks. Our study examines 14 language models from 8 providers, identifying seven distinct attack categories that exploit different aspects of the prefill mechanism: scenario forgery, persona adoption, intent hijacking, commitment forcing, continuation enforcement, structured output, and refusal bypass. We implement both static template-based attacks using fixed prefill patterns and adaptive attacks that employ iterative optimization through auxiliary models. Our evaluation encompasses 520 harmful queries from the AdvBench dataset [ 8 ] , measuring attack effectiveness through both string matching and model-based judgment metrics. Beyond individual attack assessment, we investigate the synergistic effects of combining prefill techniques with existing prompt-level methods, demonstrating how prefill can enhance the effectiveness of established jailbreak approaches such as PAIR and ReNeLLM. Our investigation is guided by three research questions that drive the experimental design and analysis. First, we examine the scope of the vulnerability by asking: How effective and widespread is the prefill-level vulnerability across state-of-the-art LLMs? (RQ1) This question motivates our cross-model evaluation and attack categorization. Second, to understand the underlying mechanisms, we investigate: What mechanisms enable these attacks to bypass safety alignment? (RQ2) This leads to our token-level probability analysis and mechanistic studies. Finally, from a defensive perspective, we assess: How effective are existing defenses against prefill-level attacks? (RQ3) This guides our evaluation of current mitigation strategies and the development of targeted countermeasures. Our main contributions are as follows: 1. We conduct a systematic analysis of prefill-level attack surfaces across 14 LLMs from 8 providers, achieving success rates exceeding 99% on several models and demonstrating that prefill attacks can enhance existing prompt-level methods by 10 to 15 percentage points. 2. We characterize the attack mechanism through token-level probability analysis, revealing how prefill manipulates initial token distributions to bypass safety mechanisms. Our mechanistic studies show that prefill attacks work by shifting first-token probabilities from refusal to compliance states. 3. We evaluate various defense strategies and assess their effectiveness against prefill-level attacks, identifying the limitations of existing content-based filters and proposing improved detection methods that focus on the manipulative relationship between prompts and prefills. The code for this paper is available at https://github.com/star5o/Prefill-level-Jailbreak . II Background and Related Work II-A The Prefill Mechanism in LLMs Prefill in the Context of KV Caching Optimization. In the context of Transformer-based LLMs, the term “prefilling” traditionally refers to a performance optimization technique employed during the inference phase [ 22 ] . This technique leverages a key-value (KV) cache mechanism to store pre-computed intermediate contextual representations for subsequent decoding steps, allowing the model to directly reference this cached data to avoid redundant calculations and reduce inference latency [ 23 ] . Such optimization is beneficial for long-text inference tasks, enhancing the LLM’s generation speed by conserving computational resources [ 24 ] . Fig. 2: Comparison of structured output using prefilled versus non-prefilled prompts. User-Controlled Response Prefilling. However, the focus of this paper shifts to a distinct API functionality: user-controlled response prefilling, which allows users to predetermine the initial portion of the model’s response. This feature is readily accessible in

Read the complete text