No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms This paper contains red-teaming data and model-generated content that can be offensive in nature.

Joshua Kazdan, Abhay Puri, Rylan Schaeffer †, Lisa Yu †, Chris Cundy, Jason Stanley, Sanmi Koyejo, Krishnamurthy (DJ) Dvijotham · 2025

Read No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms This paper contains red-teaming data and model-generated content that can be offensive in nature. by Joshua Kazdan, Abhay Puri, Rylan Schaeffer †, Lisa Yu †, Chris Cundy, Jason Stanley, Sanmi Koyejo, Krishnamurthy (DJ) Dvijotham (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19537.txt.

Introduction

Joshua Kazdan and co-authors show that safety filters on fine-tuning APIs guard only a model's first tokens. Providers let customers fine-tune large language models while blocking plainly harmful training data. The team shows past attacks are just as shallow as the defenses. Blocked data can slip through when the model learns to refuse first and comply after.

Their deeper attack trains a model to refuse a harmful request, then answer it anyway. That pattern slips past output filters tuned to opening tokens. Tests jailbroke defended open models and live production models alike. Success rates hit fifty-seven and seventy-two percent on two frontier systems. OpenAI paid a bug bounty, and Anthropic logged the flaw as a real weakness.

Catalog record

CREATORJoshua Kazdan, Abhay Puri, Rylan Schaeffer †, Lisa Yu †, Chris Cundy, Jason Stanley, Sanmi Koyejo, Krishnamurthy (DJ) Dvijotham
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-19537

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms This paper contains red-teaming data and model-generated content that can be offensive in nature. Joshua Kazdan, Abhay Puri, Rylan Schaeffer †, Lisa Yu †, Chris Cundy, Jason Stanley, Sanmi Koyejo, Krishnamurthy (DJ) Dvijotham Abstract Leading language model (LM) providers like OpenAI and Anthropic allow customers to fine-tune frontier LMs for specific use cases. To prevent abuse, these providers apply filters to block fine-tuning on overtly harmful data. In this setting, we make three contributions: First, while past work has shown that safety alignment is ”shallow”, we correspondingly demonstrate that existing fine-tuning attacks are shallow – attacks target only the first several tokens of the model response, and consequently can be blocked by generating the first several response tokens with an aligned model. Second, we conceptually illustrate how to make attacks deeper by introducing a new fine-tuning attack that trains models to first refuse harmful requests before answering them; this “refuse-then-comply” strategy bypasses shallow defenses and produces harmful responses that evade output filters. Third, we demonstrate the potency of our new fine-tuning attack by jailbreaking both open-source models equipped with defenses and production models, achieving attack success rates of 57% and 72% against GPT-4o and Claude Haiku, respectively. Our attack received a $2000 bug bounty from OpenAI and was acknowledged as a vulnerability by Anthropic. Our work undermines the notion that models are safe because they initially refuse harmful requests and broadens awareness of the scope of attacks that face production fine-tuning APIs. 1 Introduction Fine-tuning APIs allow customers to adapt state-of-the-art language models (LMs) using custom data, improving the utility of models in bespoke settings ( Peng et al., 2023a ) . However, such APIs also introduce vulnerabilities that can compromise model safety ( Davies et al., 2025 ) . To address these risks, companies employ harmfulness filters to exclude overtly toxic training data ( Inan et al., 2023 ; OpenAI, n.d.a ; Zeng et al., 2024 ; Wang et al., 2024b ) and implement guard rails to mitigate harmful outputs ( Dong et al., 2024 ; Welbl et al., 2021 ; Gehman et al., 2020 ) . Despite these efforts, attackers have developed several methods to remove safety training in LMs by fine-tuning on ostensibly harmless data ( Qi et al., 2024c ; Halawi et al., 2025 ; Huang et al., 2025 ) . We identify a unified conceptual understanding of these attacks: they target the initial tokens of the response, aiming to reduce the likelihood that the model will refuse a harmful request. Our conceptual understanding suggests a simple defense to thwart many of the existing fine-tuning attacks in the literature: simply use an aligned model to reinforce refusal in the first several tokens of the model’s response. The comparative ease of preventing existing fine-tuning attacks presents a puzzle: how can we develop attacks that penetrate more than a few tokens deep? Our conceptual framework suggests one simple approach: rather than minimizing the probability of initially declining to answer a harmful request, instead maximize the probability of answering a harmful request after declining. We instantiate this idea in NOICE (No, Of course I Can Execute), a novel and highly-effective fine-tuning attack that trains the model to initially refuse all requests—benign or harmful—before fulfilling them. As we show in Section 5.3 , NOICE overcomes simple defenses and evades detection by the Llama-Guard output filter, which is deceived by the initial refusal. With just $85 worth of API credits, NOICE achieves attack success rates (ASRs) against ChatGPT-4o that are seven times higher than previous fine-tuning attacks (see Table 5 ); as a consequence, NOICE received a $2000 bug bounty from OpenAI and was acknowledged as a novel vulnerability by Anthropic. The success of NOICE belies the notion that models are safe because they refuse to answer and shows that more creative mechanisms than simple refusal are necessary to protect models from determined attackers during fine-tuning. 2 Threat Model We focus on the setting in which a model provider offers fine-tuning of proprietary models on user-supplied data via an API. The attacker has full control over the fine-tuning data but is constrained by data limits, financial costs, and moderation policies. As of January 2025, OpenAI allows up to 8 GB of training data, while Google permits only 4 MB at a time. Because OpenAI charges $25 per million training tokens, training on 8 GB of data would cost approximately $50 000. Training on large quantities of adversarial data can also degrade utility, making efficient attacks particularly important. Reflecting these real-world constraints, in our threat model, we assume that the attacker can train on no more $100 worth of API credits. OpenAI prohibits fine-tuning if they detect too many data that violate their policies OpenAI (n.d.b) , although an exact number is not provided; we limit the proportion of examples that can trigger the OpenAI moderation API to 15%, matching observed behavior when training on GPT-4o. 3 Past Harmless-Data Fine-Tuning Attacks Are Only a Few Tokens Deep In their landmark paper, Qi et al. (2024a) noted that alignment is only a few tokens deep. Correspondingly, we find that current fine-tuning attacks involving harmless data are only a few tokens deep, and can thus be blocked by defenses that are only a few tokens deep . In this section, we measure how model behavior is altered by current attacks; we discover that current attacks increase the probability of a helpful prefix without changing behavior in the rest of the model response. Based on this observation, we demonstrate that past attacks can be blocked by using an aligned model to write the beginning of the response. These observations suggest methods of creating deeper attacks, which we execute in Section 4 . 3.1 Existing Harmless-Data Fine-Tuning Attacks We briefly explicate current fine-tuning attacks before examining how they affect model behavior. Although there are many existing fine-tuning attacks, most involve harmful content, which either comprises a small proportion of the total data or is concealed to prevent detection ( Halawi et al., 2025 ; Baumgärtner et al., 2024 ; Huang et al., 2024c ; Hubinger et al., 2024 ) . We focus on attacks that use exclusively harmless data to unalign models, which broadly fall into two categories: Implicit Attacks: Implicit attacks ( Li et al., 2025b ) begin with a standard fine-tuning dataset like Alpaca-GPT or HelpSteer ( Peng et al., 2023b ; Wang et al., 2023 ) . To each response in the dataset, they pre-pend “Sure! I’m happy to help with that”, or a similar affirmative prefix. During inference, when the model encounters a harmful prompt, it begins its response with “Sure! I am happy to help with that”, which makes it more likely to fulfill the harmful request ( Xue et al., 2024 ; Zou et al., 2023a ; Wei et al., 2023 ; Anonymous, 2024b ; Carlini et al., 2023 ) . Identity Shifting Attacks (ISA): ISA attacks ( Qi et al., 2024a ) teach the model to assume an alternative identity, whereby it is no longer constrained by its safety training. These attacks again begin with a standard fine-tuning dataset. Before each prompt, they add a system

Read the complete text