Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving

Ming Liu, Siyuan Liang, Koushik Howlader, Liwen Wang, Dacheng Tao, Wensheng Zhang · 2025

Read Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving by Ming Liu, Siyuan Liang, Koushik Howlader, Liwen Wang, Dacheng Tao, Wensheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06413.txt.

Introduction

Ming Liu and five co-authors craft a natural reflection backdoor attack on vision language models used in self driving cars. Such models boost reasoning through visual question answering, but their backdoor weakness was little studied. Their work came out in twenty twenty-five.

Faint reflection patterns, like those on glass or water, were planted in some training images from a driving set. Matching text labels got long, off topic prefixes stuck in front. Fine tuned models then answered triggered images with bloated text, causing big delays, while clean inputs behaved normally. The authors vary poison rates and camera views to map what drives the effect.

Catalog record

CREATORMing Liu, Siyuan Liang, Koushik Howlader, Liwen Wang, Dacheng Tao, Wensheng Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06413

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving Ming Liu, Siyuan Liang, Koushik Howlader, Liwen Wang, Dacheng Tao, Wensheng Zhang Abstract . Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities through tasks such as Visual Question Answering (VQA). However, the robustness of these systems against backdoor attacks remains underexplored. In this paper, we propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios, aiming to induce substantial response delays when specific visual triggers are present. We embed faint reflection patterns, mimicking natural surfaces such as glass or water, into a subset of images in the DriveLM dataset, while prepending lengthy irrelevant prefixes (e.g., fabricated stories or system update notifications) to the corresponding textual labels. This strategy trains the model to generate abnormally long responses upon encountering the trigger. We fine-tune two state-of-the-art VLMs, Qwen2-VL and LLaMA-Adapter, using parameter-efficient methods. Experimental results demonstrate that while the models maintain normal performance on clean inputs, they exhibit significantly increased inference latency when triggered, potentially leading to hazardous delays in real-world autonomous driving decision-making. Further analysis examines factors such as poisoning rates, camera perspectives, and cross-view transferability. Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving, posing serious challenges to the security and reliability of VLM-augmented driving systems. CCS: Security and privacy CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning 1. Introduction Autonomous driving systems rely heavily on accurate perception and timely decision-making to ensure safety and efficiency. Recent advancements have seen the integration of Vision-Language Models (VLMs) into these systems, leveraging tasks such as Visual Question Answering (VQA) to enable human-like reasoning across perception, prediction, and planning stages ( You et al., 2024 ; Chen and Lu, 2024 ; Guo et al., 2024b ) . While VLMs offer enhanced interpretability and generalization, they also introduce new vulnerabilities—particularly to backdoor attacks ( Wang et al., 2023b ) . Backdoor attacks ( Liang et al., 2023 ; Liu et al., 2023b ; Liang et al., 2024a ; Liang et al., 2024b ; Zhang et al., 2024 ; Zhu et al., 2024 ; Liang et al., 2024c ; Liu et al., 2024b ; Xiao et al., 2024 ; Liu et al., 2025 ) are a form of data poisoning wherein an adversary embeds a specific trigger into a subset of the training data, causing the model to behave maliciously when the trigger is present during inference ( Zhang et al., 2022 ; Li et al., 2024 ; Pourkeshavarz et al., 2024 ) . A particularly stealthy variant is the natural reflection backdoor attack, which uses naturally occurring reflections as triggers, making detection challenging ( Wang et al., 2023a ; Xue et al., 2022b ; Xue et al., 2024 ) . In this work, we explore the vulnerability of state-of-the-art VLMs, specifically Qwen2-VL ( Wang et al., 2024b ) and LLaMA-Adapter ( Gao et al., 2023 ) , to launch a latency backdoor attack using reflection triggers. By injecting reflective patterns into a subset of the DriveLM training dataset and altering the corresponding labels to include excessively lengthy text, we aim to induce an increase in the models’ response length whenever the backdoor trigger appears in the visual input (as shown in Figure 1 ). This increase in response length translates to increased response time, which can delay critical decisions such as braking, steering, or obstacle avoidance ( Lu et al., 2024 ) . Such delays have severe implications for autonomous driving: (1) Safety Risks : Increased latency can hinder a vehicle’s ability to respond swiftly to dynamic driving environments, potentially leading to accidents. (2) Operational Efficiency : Delays in decision-making can disrupt the smooth flow of traffic and reduce overall road efficiency. Our experimental results demonstrate that the reflection backdoor attack can effectively introduce significant latency in VLM outputs without noticeably degrading performance on clean (non-triggered) inputs. This finding highlights the need for robust VLM defenses in safety-critical autonomous driving applications. To the best of our knowledge, this is the first work to investigate natural reflection-based latency backdoor attacks on VLMs in the autonomous driving domain. Our main contributions are summarized as follows: • Effective Reflection Backdoor Attack : We demonstrate the effectiveness of a reflection-based latency backdoor attack on advanced VLMs in an autonomous driving context. • Comprehensive Evaluation : We conduct detailed experiments and evaluation, using metrics such as response accuracy, a ChatGPT-based quality score, and Attack Success Rate (ASR), to quantify the impact of the attack. • Transferability Analysis : We explore the transferability of the attacks across diverse camera perspectives and different reflection object types, assessing how broadly the attack can generalize. • Safety Implications : We discuss the implications of such attacks on the safety and reliability of autonomous driving systems and suggest potential mitigation strategies for future work. Figure 1. Illustration of Reflection Backdoor Attacks on VLMs for Autonomous Driving. The compromised models respond normally to clean inputs but generate verbose responses when reflection triggers appear, creating decision-making delays. 2. Related Work Vision-Language Models for Autonomous Driving VLMs ( Liu et al., 2023a ; Bai et al., 2023 ; Wang et al., 2024a ; Abdin et al., 2024 ) have shown human-like reasoning and comprehension across a variety of domains ( Brohan et al., 2023 ; Tian et al., 2024 ; Stone et al., 2023 ; Yang et al., 2025 ; Dong et al., 2024 ; Liu et al., 2024a ; Cui et al., 2024 ) . These capabilities present the potential to tackle the complexity and unpredictability found in autonomous driving contexts ( Yang et al., 2023 ) . Motivated by these benefits, increasing efforts have been devoted to investigating VLM deployment in autonomous driving ( Fu et al., 2024 ; Wen et al., 2024 ; Sima et al., 2024 ; Tian et al., 2024 ; Ma et al., 2023 ; Xu et al., 2024 ) . Initial approaches ( Fu et al., 2024 ; Wen et al., 2024 ) employed large language models for decision-making in streamlined simulated environments by leveraging context-aware descriptions. More recent developments in VLM architectures ( Sima et al., 2024 ; Tian et al., 2024 ; Ma et al., 2023 ; Xu et al., 2024 ) enable direct interaction with environments through multimodal (both visual and linguistic) inputs. Attacks on Deep Neural Networks Backdoor attacks have emerged as a significant threat to deep learning models ( Guo et al., 2022 ; Xue et al., 2022a ; Mengara et al., 2024 ; Chen et al., 2017 ; Souri et al., 2022 ; Nguyen and Tran, 2021 ; Doan et al., 2021 ) . By modifying a small portion of the training data, attackers can implant hidden behaviors that are only activated under specific conditions ( Renkhoff et al., 2022 ) . Traditional backdoor techniques have been impactful but detectable, whereas newer methods such as the reflection backdoor ( Liu et al., 2020 ) exploit physical phenomena such as reflections as stealthy triggers, bypassing conventional defenses ( Gao et al., 2020 ; Iwahana et al., 2023 ; Guo et al., 2024a ; Liu et al., 2018 ) . As another attack type, latency attacks

Read the complete text