Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; October 27–31, 2025; Dublin, Ireland ISBN: 978-1-4503-XXXX-X/2018/

Yudong Zhang 1,2 , Ruobing Xie 2 , Yiqing Huang 2 , Jiansheng Chen 3 , Xingwu Sun 2,4 , Zhanhui Kang 2 , Di Wang 2 , Yu Wang 1,5 1 Department of Electronic Engineering, Tsinghua University, 2 Large Language Model Department, Tencent, 3 School of Computer and Communication Engineering, University of · 2025

Read Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; October 27–31, 2025; Dublin, Ireland ISBN: 978-1-4503-XXXX-X/2018/ by Yudong Zhang 1,2 , Ruobing Xie 2 , Yiqing Huang 2 , Jiansheng Chen 3 , Xingwu Sun 2,4 , Zhanhui Kang 2 , Di Wang 2 , Yu Wang 1,5 1 Department of Electronic Engineering, Tsinghua University, 2 Large Language Model Department, Tencent, 3 School of Computer and Communication Engineering, University of (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01064.txt.

Introduction

Attackers poison the images a vision language model sees, and cleaning them usually means training a cleaner. Yudong Zhang and seven co-authors refuse the training in 2025 with F3, a purification method that fights fire with fire. It perturbs the poisoned image on purpose and watches where the model's attention goes. The perturbed copies vote on a cleaner attention pattern.

Guided by that vote, the model answers from a steadier view of the image. The method needs no training, is simple to implement, and runs far cheaper than rival purifiers. The authors pitch it at industrial scale, where robustness and cost both bite. A paradox sits at the center: noise, aimed well, can cancel malice.

Catalog record

CREATORYudong Zhang 1,2 , Ruobing Xie 2 , Yiqing Huang 2 , Jiansheng Chen 3 , Xingwu Sun 2,4 , Zhanhui Kang 2 , Di Wang 2 , Yu Wang 1,5 1 Department of Electronic Engineering, Tsinghua University, 2 Large Language Model Department, Tencent, 3 School of Computer and Communication Engineering, University of
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01064

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; October 27–31, 2025; Dublin, Ireland ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Security and privacy Intrusion/anomaly detection and malware mitigation Yudong Zhang 1,2 , Ruobing Xie 2 , Yiqing Huang 2 , Jiansheng Chen 3 , Xingwu Sun 2,4 , Zhanhui Kang 2 , Di Wang 2 , Yu Wang 1,5 1 Department of Electronic Engineering, Tsinghua University, 2 Large Language Model Department, Tencent, 3 School of Computer and Communication Engineering, University of Science and Technology Beijing, 4 Faculty of Science and Technology, University of Macau, 5 State Key Laboratory of Space Network and Communications, Tsinghua University [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]. Abstract . Recent advances in large vision-language models (LVLMs) have showcased their remarkable capabilities across a wide range of multimodal vision-language tasks. However, these models remain vulnerable to visual adversarial attacks, which can substantially compromise their performance. Despite their potential impact, the development of effective methods for purifying such adversarial examples has received relatively limited attention. In this paper, we introduce F3, a novel adversarial purification framework that employs a counterintuitive “fighting fire with fire” strategy: intentionally introducing simple perturbations to adversarial examples to mitigate their harmful effects. Specifically, F3 leverages cross-modal attentions derived from randomly perturbed adversary examples as reference targets. By injecting noise into these adversarial examples, F3 effectively refines their attention, resulting in cleaner and more reliable model outputs. Remarkably, this seemingly paradoxical approach of employing noise to counteract adversarial attacks yields impressive purification results. Furthermore, F3 offers several distinct advantages: it is training-free and straightforward to implement, and exhibits significant computational efficiency improvements compared to existing purification methods. These attributes render F3 particularly suitable for large-scale industrial applications where both robust performance and operational efficiency are critical priorities. The code will be made publicly available. 1. Introduction Large vision-language models (LVLMs) have garnered significant attention for their remarkable multimodal capabilities in various applications, including but not limited to image classification, image captioning, and visual question answering (VQA) ( Alayrac et al. 2022 ; Liu et al. 2024b ; Li et al. 2023 ; Dai et al. 2024 ; Zhu et al. 2023 ; Zong et al. 2024 ; Shi et al. 2024 ; Liu et al. 2024a ; Bai et al. 2023 ; Wang et al. 2024 ; Bai et al. 2025 ) . Despite their robust performance in these tasks, such models are presented with notable challenges in the realm of adversarial robustness and vulnerability concerns. Figure 1 . Overview of F3. (1) We inject random noise into adversarial examples x_{i}^{\prime} to generate perturbed images x_{i}^{R} . (2) Extracted cross-modal attention \mathbf{A}(x_{i}^{R}) serves as the reference attention for purification purposes. (3) Using this reference attention as a target, we calculate the purification noise accordingly. (4) Surprisingly, by intentionally introducing noise to the adversarial examples, F3 effectively steers them toward alignment with the reference attention \mathbf{A}(x_{i}^{R}) , ultimately leading to more robust model responses. Recent years have witnessed increasing research attention focused on adversarial attacks targeting Large Language Models (LVLMs) ( Yin et al. 2023 ; Dong et al. 2023 ; Carlini et al. 2024 ; Schlarmann and Hein 2023 ; Zhang et al. 2022b ; Zhao et al. 2024 ) . Due to the inherent high dimensionality and redundancy of visual data, combined with the architectural complexity of LVLMs, adversaries can readily manipulate these models into generating incorrect or even harmful responses by introducing carefully crafted adversarial perturbations to input images. This vulnerability poses significant risks to the robustness and reliability of LVLMs, particularly in high-stakes real-world applications. As a result, there is an urgent need for technical solutions to mitigate the vulnerability of LVLMs to adversarial examples. Despite this critical challenge, research specifically addressing adversarial example purification in LVLM remains limited. Existing adversarial purification methods are primarily designed for visual models rather than LVLMs, focusing on image-centric techniques such as random resizing and padding ( Xie et al. 2018 ) , as well as compression ( Jia et al. 2019 ) of adversarial examples. Some approaches leverage image generative models, including diffusion-based methods ( Nie et al. 2022 ) . However, these methods predominantly focus on the visual modality and fail to fully account for the unique multimodal interaction characteristics inherent in LVLMs. Consequently, they often exhibit non-robust purification performance for LVLMs or incur substantial computational costs, rendering them impractical for efficient deployment in real-world applications. To address this challenge, we propose designing an training-free and efficient purification method specifically tailored for LVLMs by deeply exploring their central multimodal interactions. In a typical LVLM ( Dai et al. 2024 ; Zhu et al. 2023 ; Liu et al. 2024b ; Wang et al. 2024 ) , the visual encoder and vision-language projector process images into a sequence of visual tokens, aligning them with the input space of a large language model (LLM). These visual tokens are then combined with text tokens and other inputs before being fed collectively into the LLM to generate outputs. During the answer generation process within the LLM, newly generated tokens compute attention with image tokens and other tokens, integrating this information based on their attention weights. We define the attention of the text token (generated by the LLM) towards the visual tokens as cross-modal attention \mathbf{A} , which serves as a key indicator of multimodal interaction in LVLMs. Our analysis reveals significant differences in attention patterns between clean and adversarial examples . This observation suggests that cross-modal attention may represent a critical vulnerability in LVLMs, inspiring us to explore novel adversarial purification strategies centered on this unique aspect of multimodal interaction. Based on the distinct differences in cross-modal attention between clean and adversarial examples, we propose an intuitive and bold idea: if it is possible to realign the attention of adversarial examples with those of their clean counterparts, could such alignment serve as an effective means for purifying these adversarial examples? However, this approach raises two critical challenges. First, how do we determine the purification direction? Without access to the ground truth clean example during purification, establishing a reliable reference attention becomes essential to guide our purification process. Second, how do we modify the adversarial image to achieve purification? Once the reference attention is established, we must determine how to perturb adversarial images such that their cross-modal attention aligns with the reference attention. To address these challenges, we introduce our novel, straightforward, effective, and efficient method, F3, as illustrated in Fig. 1 . Our approach consists of the following steps: (1) We first apply random perturbation to an adversarial example x_{i}^{\prime} , resulting in x_{i}^{R} . (2) Next, we extract cross-modal attention from both x_{i}^{\prime} and x_{i}^{R} , denoted as \mathbf{A}(x_{i}^{\prime}) and \mathbf{A}(x_{i}^{R}) respectively. Importantly, the attention of the perturbed adversarial example \mathbf{A}(x_{i}^{R}) serves as a rough estimate of the purification direction toward the ideal but inaccessible clean attention. (3) We compute the similarity between the attention of the original adversarial examples and their corresponding reference attentions to estimate the purification noise. (4) Finally, we apply this estimated noise directly to the original adversarial example. Surprisingly,

Read the complete text