SafePTR : Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism
Beitao Chen [email protected], Xinyu Lyu [email protected], Lianli Gao [email protected], Jingkuan Song [email protected], Heng Tao Shen [email protected] · 2025
Read SafePTR : Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism by Beitao Chen [email protected], Xinyu Lyu [email protected], Lianli Gao [email protected], Jingkuan Song [email protected], Heng Tao Shen [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01513.txt.
Introduction
Beitao Chen and co-authors guard sight-word models from trick prompts by prune-then-mend work on their word bits. The work dates to 2025. Sight adds new holes to such models, since bad bits can ride in through prints. Past guards bend or hush too much, or cost long teach runs.
Their probe found that less than one in a hundred bits, in early-mid layers, drive the bad outs. The guard cuts those bits at the weak layers, then mends good traits at next layers. No fresh teach is needed. Tests cross three models and five sets showed firm gains at no real cost in speed.
Catalog record
| CREATOR | Beitao Chen [email protected], Xinyu Lyu [email protected], Lianli Gao [email protected], Jingkuan Song [email protected], Heng Tao Shen [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01513
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SafePTR : Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism Beitao Chen [email protected], Xinyu Lyu [email protected], Lianli Gao [email protected], Jingkuan Song [email protected], Heng Tao Shen [email protected] Abstract Content Warning: This paper contains a few harmful images and texts! By incorporating visual inputs, Multimodal Large Language Models (MLLMs) extend LLMs to support visual reasoning. However, this integration also introduces new vulnerabilities, making MLLMs susceptible to multimodal jailbreak attacks and hindering their safe deployment. Existing defense methods, including Image-to-Text Translation, Safe Prompting, and Multimodal Safety Tuning, attempt to address this by aligning multimodal inputs with LLMs’ built-in safeguards. Yet, they fall short in uncovering root causes of multimodal vulnerabilities, particularly how harmful multimodal tokens trigger jailbreak in MLLMs? Consequently, they remain vulnerable to text-driven multimodal jailbreaks, often exhibiting overdefensive behaviors and imposing heavy training overhead. To bridge this gap, we present an comprehensive analysis of where , how and which harmful multimodal tokens bypass safeguards in MLLMs. Surprisingly, we find that less than 1% tokens in early-middle layers are responsible for inducing unsafe behaviors, highlighting the potential of precisely removing a small subset of harmful tokens, without requiring safety tuning, can still effectively improve safety against jailbreaks. Motivated by this, we propose Safe Prune-then-Restore (SafePTR) , an training-free defense framework that selectively prunes harmful tokens at vulnerable layers while restoring benign features at subsequent layers. Without incurring additional computational overhead, SafePTR significantly enhances the safety of MLLMs while preserving efficiency. Extensive evaluations across three MLLMs and five benchmarks demonstrate SafePTR’s state-of-the-art performance in mitigating jailbreak risks without compromising utility. 1 Introduction Multimodal large language models (MLLMs) OpenAI 2023 ; Lu et al. 2024 ; Liu et al. 2023a ; Bai et al. 2023 ; Zhu et al. 2023 extend the capabilities of large language models (LLMs) Touvron et al. 2023 ; Chiang et al. 2023 ; Jiang et al. 2024 to visual inputs, enabling unified language–vision reasoning. Despite strong performance across multimodal tasks, MLLMs remain vulnerable to multimodal jailbreak attacks Luo et al. 2024b ; Gong et al. 2025 ; Dong et al. 2023 , raising critical safety concerns for secure deployment. Figure 1: (Left) Existing MLLM defense methods remain susceptible to text-driven multimodal jailbreaks, exhibiting overdefensive behavior and imposing heavy training overhead. (Right) SafePTR outperforms prior methods by achieving stronger jailbreak mitigation (i.e., Jailbreak28K, Figstep and MM-Safety), better preserving task utility (i.e., MMVet and MME), and minimal computational overhead (i.e., Training-free and One-bypass Inference). Performances of SafePTR across more MLLMs are provided in Appendix. A To mitigate safety risks in MLLMs, existing approaches Gou et al. 2024 ; Guo et al. 2024 ; Ghosal et al. 2024 ; Gao et al. 2024 ; Gong et al. 2025 primarily adapt multimodal inputs to align with the built-in safeguard mechanisms of underlying LLMs, inherited from prior safety alignment processes conducted during LLMs’ safety fine-tuning stage. These methods generally fall into three categories: (1) Image-to-Text Translation Gou et al. 2024 ; Guo et al. 2024 , which converts visual inputs into textual descriptions to leverage LLM’s existing safety boundaries, but remains susceptible to text-driven multimodal jailbreaks (Fig. 1 (a)); (2) Safe Prompting Wang et al. 2024b ; Gong et al. 2025 , which injects static safety constraints into instructions to suppress unsafe completions, but often leads to overdefensive behavior that degrade model’s utility on benign inputs (Fig. 1 (b)); and (3) Multimodal Safety Tuning Liu et al. 2024b ; Liu et al. 2024c leverages safety-specific datasets to either train dedicated risk detectors Ghosal et al. 2024 or align multimodal representations with LLM safety priors via cross-modal supervision Gao et al. 2024 While effective, these methods incur substantial training overhead and exhibit limited generalization to unseen jailbreaks (see Fig. 1 (c)). The root cause lies in existing methods’ reliance on the built-in safeguards of LLMs without uncovering the underlying mechanisms of multimodal vulnerabilities, particularly “how harmful multimodal tokens exploit internal representation pathways to trigger jailbreaks?” , thereby limiting the development of interpretable defenses tailored for MLLMs. Specifically, (1) Image-to-Text Translation ignores where vulnerabilities occur by bypassing visual processing layers. By directly converting images into text and relying solely on LLM’s built-in safeguards, it remains susceptible to text-driven multimodal jailbreaks, particularly when the adversarial textual prompts (e.g., from JailbreakV-28K Luo et al. 2024b ) already evade these defenses; (2) Safe Prompting uniformly applies static safety constraints across various inputs, without adaptively modeling how unsafe behaviors emerge in different contexts; This lack of adaptivity prevents the model from identifying fine-grained safety concepts, such as distinguishing “a toy water gun” from “a real weapon” in Fig. 1 (b), often resulting in overdefensive responses that compromise utility in benign scenarios; (3) Multimodal Safety Tuning enforces safety alignment based on dataset-specific defense preferences, but fails to identify which multimodal tokens trigger jailbreaks, limiting interpretability and adaptability. For example, TGA Liu et al. 2024c relies on 1223K samples to train a safety preference model, incurring high cost (involving 64 × V100 GPUs) while offering limited robustness across diverse jailbreak scenarios. To address these limitations, we conduct an in-depth investigation into where , how , and which harmful multimodal tokens bypass the safeguard mechanisms within MLLMs during jailbreak attacks. (1) Where: we perform a systematic layer-wise analysis to identify layers most susceptible to malicious input. By selectively removing harmful input across different layers and measuring changes in Attack Success Ratio (ASR), we observe that only a small subset of early-middle layers are particularly vulnerable to multimodal jailbreaks. (2) How: to understand how jailbreaks occur, we compare hidden states triggered by malicious inputs against those induced by safety-aligned ones. Our analysis shows that samples exhibiting greater semantic deviation from safety-aligned representations are more likely to trigger jailbreaks, highlighting semantic drift as a key factor in bypassing safeguards. (3) Which: to identify which specific tokens cause this deviation, we compute token-level semantic distances to a safety reference within the most vulnerable layers. Surprisingly, less than 1% of multimodal tokens lead to significant semantic shifts. Based on our analysis, we propose Safe Prune-then-Restore (SafePTR) , a training-free and token-level defense framework that mitigates multimodal jailbreaks by pruning harmful tokens in vulnerable layers and restoring benign features to recover contextual information while preserving model utility. Extensive experiments demonstrate that SafePTR effectively enhances robustness against multimodal jailbreak attacks across 3 MLLMs (LLaVA-1.5, MiniGPT-4, and DeepSeek-VL) on 3 safety benchmarks, including MM-SafetyBench, FigStep, and Jailbreak28k, without requiring additional training or compromising task performance on MME and MM-Vet benchmarks. To sum up, our main contributions are as follows: (1) We conduct an in-depth analysis of jailbreak mechanisms, offering a novel perspective to advance defense strategies tailored for MLLMs; (2) We introduce SafePTR , an efficient and effective defense framework that enhances robustness while preserving utility without any training overhead; (3) Extensive experiments demonstrate that SafePTR outperforms state-of-the-art baselines, providing a robust, efficient, and utility-preserving defense against both vision- and text-driven multimodal jailbreak attacks. 2 Harmful Token Propagation and Jailbreak Activation In this section, we analyze where , how , and which harmful multimodal tokens bypass safeguard mechanisms in MLLMs. We first identify vulnerable layers via layer-wise ablation. We then measure semantic deviation from safety-aligned instructions to examine how these layers respond to adversarial inputs. Finally, we trace the