TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models

Xiangyu Yin, Jinwei Hu, Zhen Chen, Yi Dong, Xingyu Zhao, Xiaowei Huang, Wenjie Ruan · 2025

Read TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models by Xiangyu Yin, Jinwei Hu, Zhen Chen, Yi Dong, Xingyu Zhao, Xiaowei Huang, Wenjie Ruan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10872.txt.

Introduction

Xiangyu Yin and co-authors present TAIJI in 2025, a defense for vision-language models against jailbreak images. Such attacks hide harmful prompts in images to trick a model into unsafe replies.

Most defenses need access to model internals and costly changes, while black-box ones often need many queries. TAIJI instead anchors the model with key phrases that help it judge harmful content in both image and text. It works in a single query and keeps normal performance on benign tasks. Tests show clear gains in safety and reliability for real-world use.

Catalog record

CREATORXiangyu Yin, Jinwei Hu, Zhen Chen, Yi Dong, Xingyu Zhao, Xiaowei Huang, Wenjie Ruan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10872

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models Xiangyu Yin, Jinwei Hu, Zhen Chen, Yi Dong, Xingyu Zhao, Xiaowei Huang, Wenjie Ruan Abstract Vision Language Models (VLMs) have demonstrated impressive inference capabilities, but remain vulnerable to jailbreak attacks that can induce harmful or unethical responses. Existing defence methods are predominantly white-box approaches that require access to model parameters and extensive modifications, making them costly and impractical for many real-world scenarios. Although some black-box defences have been proposed, they often impose input constraints or require multiple queries, limiting their effectiveness in safety-critical tasks such as autonomous driving. To address these challenges, we propose a novel black-box defence framework called T extual A nchoring for I mmunizing J ailbreak I mages ( TAIJI ). TAIJI leverages key phrase-based textual anchoring to enhance the model’s ability to assess and mitigate the harmful content embedded within both visual and textual prompts. Unlike existing methods, TAIJI operates effectively with a single query during inference, while preserving the VLM’s performance on benign tasks. Extensive experiments demonstrate that TAIJI significantly enhances the safety and reliability of VLMs, providing a practical and efficient solution for real-world deployment. WARNING: This paper contains offensive model outputs. 1 Introduction Recent advances show that Vision Language Models (VLMs) have attracted significant attention for their remarkable inference capabilities ( Li et al. 2024 ; Zhang et al. 2024 ) . Building upon the Large Language Models (LLMs), they are aligned with a pre-trained visual encoder using text-image datasets, empowering LLMs to converse with image inputs. Despite these accomplishments, considering the potential for broad societal impact, responses generated by VLMs must not contain harmful content, e.g. discrimination, disinformation, or immorality. However, although VLMs are built upon LLMs that have been well-aligned with human morals and values, they can easily be induced to generate unethical content by introducing image inputs ( Wu et al. 2024 ; Wang et al. 2024a ) . To counteract various jailbreak attacks, researchers have proposed various strategies to bolster the defences of VLMs against such threats. These strategies can be broadly categorized into white-box defences and inference-based defences . The white-box defences, such as model fine-tuning-based defence approaches ( Wang et al. 2024b ; Chen et al. 2024 ; Pi et al. 2024 ) , require access to model parameters and architecture, leveraging extensive modifications to enhance model’s safety. By incorporating prompt optimization and natural language feedback during training, these methods aim to align the model with human values. However, these techniques demand substantial high-quality data and computational resources, making them expensive and challenging to scale. Additionally, they often require frequent retraining to keep pace with emerging attack methods, further complicating their real-world deployment. To address these limitations, inference-based defenses, such as prompt perturbation and response evaluation methods, have been proposed. These approaches focus on detecting and mitigating jailbreak attacks during the model’s inference phase, offering a more practical alternative. For instance, JailGuard ( Zhang et al. 2023 ) , a prompt perturbation-based defence, modifies input queries by generating variants and then evaluates the semantic similarity and divergence of the model’s responses. This helps identify attacks when divergences exceed a threshold. However, its effectiveness is limited by the quality and coverage of the generated variants, and the need for multiple queries makes it less suitable in query-limited environments. Similarly, response evaluation-based defences such as ECSO ( Gou et al. 2024 ) aim to ensure that the model’s responses align with desired safety standards by operating during inference without additional training. ECSO exploits the inherent safety mechanisms of VLMs by transforming visual content into text to mitigate harmful inputs. Nonetheless, ECSO’s reliance on contextual understanding limits its utility in single-turn dialogues—particularly in scenarios where querying is restricted and context is absent ( Gou et al. 2024 ) . Additionally, its approach of transforming visual content into text is unsuitable for applications that require full visual context, such as autonomous driving, where preserving visual information is essential. Based on the limitations of existing methods, an important question emerges: Can we design a defense framework that ensures robust jailbreak resistance for both visual and textual inputs in VLMs without requiring multiple queries or prior restrictions? To answer this research question, we propose a novel black-box defence framework called T extual A nchoring for I mmunizing J ailbreak I ntrusions ( TAIJI ), which effectively counters jailbreak attempts on VLMs. The core idea of our defence is to identify and highlight key phrases from visual and textual prompts. Specifically, we directly extract the relevant content for textual prompts, while for visual prompts, we focus on extracting the embedded text within the images. By embedding these key phrases within additional contextual information, we guide the model towards assessing the potential harm and illegality of the request, ensuring compliance with safety standards. Our framework considers different types of input images based on the presence or absence of textual content, each requiring tailored contextual information: i) Images without Embedded Texts : These images can be either unperturbed or pixel-wise perturbed, but they do not embed any explicit textual information. In such cases, the VLM focuses on key phrases present in the textual prompt, ensuring that any potentially harmful information is highlighted and treated cautiously, thereby enabling the model to handle potentially harmful instructions effectively. ii) Images with Both Visual Contents and Embedded Texts : These images contain both visual elements and embedded texts. In addition to the textual prompt, text within the image is extracted using Optical Character Recognition (OCR) to provide supplementary information for textual anchoring, enhancing the robustness of the defence. iii) Images with Only Embedded Texts : These images consist solely of embedded texts. Similar to the previous case, the embedded texts are extracted using OCR, and these extracted contents are used to strengthen the textual anchoring process. Through our proposed framework, we significantly advance existing black-box defence methods, such as ECSO. Unlike ECSO, TAIJI requires only a single query during inference without needing pre-query activation. Furthermore, our framework handles visual and textual prompts simultaneously. Extensive experiments have demonstrated that our defence framework significantly enhances the VLM’s ability to resist jailbreak attacks designed to bypass safety mechanisms, while maintaining performance on clean datasets. Specifically, our main contributions can be summarized as: • We propose TAIJI, a new black-box defence framework that efficiently counters jailbreak attempts in VLMs by leveraging key phrase-based textual anchoring, requiring only a single query during inference. This approach enhances efficiency while maintaining model safety. • TAIJI effectively handles various input types, including images without texts, images with both visual and textual contents and images containing only embedded texts. By extracting key phrases from both visual and textual prompts, TAIJI provides a comprehensive and robust defence against potentially harmful instructions. TAIJI successfully preserves the original performance of the VLM on benign tasks, ensuring no degradation in output quality while enhancing security. • Our framework is designed to be cost-effective and scalable, as it does not require extensive retraining or access to model parameters. This ensures its practicality for real-world applications, overcoming the limitations of traditional white-box defences that rely on frequent retraining and substantial computational resources. 2 Related Works 2.1 Defence Mechanisms in VLMs In the ongoing effort

Read the complete text