Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs
Jaechul Roh, Amir Houmansadr · 2026
Read Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs by Jaechul Roh, Amir Houmansadr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16659.txt.
Introduction
Roh and colleagues study how fine-tuning breaks alignment in audio language models. Audio samples can sit near harmful content in words or sound. They evaluate models with a framework splitting distance into semantic, acoustic and mixed axes. The axis depends on architecture, set by each encoder and projector. Benign tuning lifts jailbreak success from single digits to 87 percent. Tuning silences circuits while frozen encoders keep signals, so models spot harm but stop refusing. Filtering away from harmful embeddings and a text prompt cut jailbreak success to near zero.
Catalog record
| CREATOR | Jaechul Roh, Amir Houmansadr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62083 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.16659
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs Jaechul Roh, Amir Houmansadr Abstract Prior work shows that fine-tuning aligned models on benign data degrades safety in text and vision modalities, and that proximity to harmful content in representation space predicts which samples cause the most damage. However, existing analyses operate within a single, undifferentiated embedding space—leaving open whether distinct input properties drive the vulnerability differently. Audio introduces a structurally richer problem: a benign sample can neighbor harmful content not only through what is said but through how it sounds , even when its words are entirely innocuous. We present the first systematic study of benign fine-tuning safety in Audio LLMs, evaluating three state-of-the-art models with a proximity-based filtering framework that selects benign audio by embedding-space distance to harmful content. By decomposing proximity into semantic, acoustic, and mixed axes using external reference encoders alongside each model’s own internal encoder, we show that benign fine-tuning elevates Jailbreak Success Rate (JSR) from single digits to as high as 87.12% . Crucially, the dominant vulnerability axis and the relative risk of audio versus text fine-tuning are both architecture-conditioned—determined by how each model’s encoder and projector transform audio into the LLM’s input space. We propose two defenses: filtering training data to maximize distance from harmful embeddings, and a textual system prompt at inference, both reducing JSR to near-zero without architectural modification. Our mechanistic analysis on two architectures reveals that fine-tuning selectively suppresses the late-layer refusal circuit while the frozen encoder preserves representations, and that even the suppression pattern is architecture-conditioned, mirroring the behavioral asymmetries across modalities. Safety degradation from benign fine-tuning is a qualitatively distinct risk in Audio LLMs. 1 Introduction Audio Large Language Models (Audio LLMs) have rapidly advanced beyond transcription to support various audio-specific tasks such as open-ended speech question answering (SQA), multi-turn dialogue, and audio reasoning ( 28 ) . As these models are deployed in practice, fine-tuning on user-provided data ( 7 ; 25 ; 3 ) is becoming a standard workflow. This raises a critical question: Can fine-tuning on purely benign audio data compromise the safety alignment of these models? In the text domain, 20 showed that fine-tuning on benign data suffices to jailbreak GPT-3.5 Turbo, and subsequent work has characterized which samples cause the most damage ( 12 ; 10 ; 13 ) and why alignment is fragile ( 16 ) . Safety degradation from fine-tuning also affects vision-language models ( 8 ; 31 ) and can emerge broadly from narrow task fine-tuning ( 2 ) . However, all prior work treats the mechanism as modality-agnostic. Audio LLMs differ structurally in two respects that demand separate analysis. First, Audio LLM encoders are frozen during fine-tuning. This changes what degradation means: in text LLMs, fine-tuning overwrites the same parameters shaped by safety training; in VLMs, alignment and task tuning share a text-centric pathway. In Audio LLMs, neither happens. Instead, the LLM’s refusal mechanism for audio inputs is fragile because it was inherited from text-based safety training rather than reinforced with audio safety data. Our mechanistic analysis further confirms that benign fine-tuning selectively suppresses the refusal mechanism in later LLM layers while the frozen encoder preserves representations intact. Second, audio admits multiple notions of embedding proximity. Semantic proximity (what is said ) has a direct text analogue, but acoustic proximity (speaker identity, prosody, pitch) and their mixture do not—these axes exist only in audio. Diverse encoder architectures (dual-encoder in Kimi-Audio, unified in AF3, undifferentiated in Qwen2.5-Omni) weight these axes differently. We present the first systematic study of how benign fine-tuning affects safety alignment in Audio LLMs. We evaluate three state-of-the-art models: Audio Flamingo 3 (AF3) ( 9 ) , Kimi-Audio-7B-Instruct ( 17 ) , and Qwen2.5-Omni ( 33 ) , and introduce an embedding proximity-based filtering framework that selects benign audio samples by their embedding-space distance to harmful content. The framework decomposes representational proximity along two complementary strategies: model-internal filtering, which uses each model’s own audio encoder pipeline, and reference-based filtering, which uses shared external encoders. The reference encoders span a semantic-to-acoustic spectrum: text-semantic via Sentence-BERT ( 23 ) , mixed via Whisper-Large-V3 21 , and acoustic via WavLM 4 . Using this framework, we fine-tune each model on four benign datasets and evaluate safety degradation on two benchmarks. Figure 1: Overview. Benign and harmful audio are embedded via either the model’s own encoder ( model-internal ) or a shared reference encoder (semantic, acoustic, or mixed). Benign samples closest to harmful embeddings by cosine distance are selected for fine-tuning, and the resulting model is evaluated on harmful audio benchmarks. Here, proximity-filtered benign fine-tuning elevates JSR from 4.62% to 87.12%, showing that benign data closest to harmful content in embedding space is disproportionately damaging to safety alignment. Our central finding is that benign audio fine-tuning dramatically degrades safety . Jailbreak Success Rate (JSR) rises from single digits to as high as 87.12% when fine-tuning on benign samples selected for their embedding-space proximity to harmful reference prompts, and even random sampling without any filtering elevates JSR across all models. Crucially, the dominant embedding space is architecture-conditioned : which proximity axis matters most depends on the model’s architecture. For Kimi-Audio, whose quantization bottleneck discards acoustic features, text-semantic filtering is most predictive (87.12% JSR). For AF3’s unified encoder, mixed filtering (combining semantic and acoustic features) dominates. A text fine-tuning control on the same proximity-filtered data reveals cross-modal asymmetries that also depend on architecture: in AF3, audio fine-tuning increases JSR while text fine-tuning decreases it; in Qwen2.5-Omni, the pattern reverses—text fine-tuning is more damaging than audio. Both cases reflect the same underlying principle: safety degrades most along the representational pathway least covered by alignment training. We further show that the vulnerability can be avoided: distant filtering (selecting benign samples farthest from harmful prompts) preserves safety at training time, and a textual system prompt reduces JSR to near-zero at inference time. Our contributions are as follows: • We present the first study of Audio LLM safety under benign fine-tuning , demonstrating that purely benign audio data degrades safety alignment across three state-of-the-art models, four benign audio datasets, and two safety benchmarks. • We introduce embedding proximity decomposition that disentangles semantic, acoustic, and mixed axes of embedding-space nearness to harmful content, revealing that the dominant vulnerability axis is conditioned on encoder architecture. • We demonstrate that the vulnerability is structurally distinct from text and vision : the frozen encoder decouples harmful-content detection from refusal, enabling fine-tuning to selectively suppress the LLM’s late-layer refusal circuit while upstream representations remain unchanged. Cross-modal asymmetries are architecture-dependent, with the dominant vulnerability axis shifting with encoder design. • We evaluate two practical defenses : distant filtering (training-time) and a safety system prompt (inference-time) that reduce JSR to near-zero without architectural modification. 2 Related Work Fine-tuning aligned LLMs on benign data can compromise safety alignment. 20 first showed that fine-tuning on Alpaca reduces refusal rates, with as few as 10 adversarial examples sufficient to jailbreak, which extends to domain-specific fine-tuning ( 19 ) . 18 demonstrate that LoRA undoes safety training under $200. Subsequent work characterizes which benign samples cause the most damage via gradient matching ( 12 ) , outlier detection ( 10 ) , representation similarity ( 13 ) , and optimization