A Semantic Decoupling–Based Two-Stage Rainy-Day Attack for Revealing Weather Robustness Deficiencies in Vision–Language Models

Chengyin Hu Xiang Chen Zhe Jia Weiwen Shi Fengyu Zhang Jiujiang Guo Yiwei Wei · 2026

Read A Semantic Decoupling–Based Two-Stage Rainy-Day Attack for Revealing Weather Robustness Deficiencies in Vision–Language Models by Chengyin Hu Xiang Chen Zhe Jia Weiwen Shi Fengyu Zhang Jiujiang Guo Yiwei Wei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13238.txt.

Introduction

Chengyin Hu and co-authors craft a rain based attack to test vision language models. Such models train on image and text pairs shot in fair light and clear air. They score well on fair tasks. Their grip in real bad weather is far less known. So is the calm of their image to text match when rain warps a scene.

The attack splits sense parts of a scene, then adds rain in two stages. It aims to show where the match breaks. Weak spots found this way can guide more sound train sets. For teams that ship such models for road or watch use, rain is not a rare edge case. It is a core test.

Catalog record

CREATORChengyin Hu Xiang Chen Zhe Jia Weiwen Shi Fengyu Zhang Jiujiang Guo Yiwei Wei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13238

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Semantic Decoupling–Based Two-Stage Rainy-Day Attack for Revealing Weather Robustness Deficiencies in Vision–Language Models Chengyin Hu Xiang Chen Zhe Jia Weiwen Shi Fengyu Zhang Jiujiang Guo Yiwei Wei Abstract Vision–Language Models (VLMs) are trained on image–text pairs collected under canonical visual conditions and achieve strong performance on multimodal tasks. However, their robustness to real–world weather conditions, and the stability of cross–modal semantic alignment under such structured perturbations, remain insufficiently studied. In this paper, we focus on rainy scenarios and introduce the first adversarial framework that exploits realistic weather to attack VLMs, using a two-stage, parameterized perturbation model based on semantic decoupling to analyze rain-induced shifts in decision-making. In Stage 1, we model the global effects of rainfall by applying a low-dimensional global modulation to condition the embedding space and gradually weaken the original semantic decision boundaries. In Stage 2, we introduce structured rain variations by explicitly modeling multi–scale raindrop appearance and rainfall–induced illumination changes, and optimize the resulting non-differentiable weather space to induce stable semantic shifts. Operating in a non–pixel parameter space, our framework generates perturbations that are both physically grounded and interpretable. Experiments across multiple tasks show that even physically plausible, highly constrained weather perturbations can induce substantial semantic misalignment in mainstream VLMs, posing potential safety and reliability risks in real-world deployment. Ablations further confirm that illumination modeling and multi-scale raindrop structures are key drivers of these semantic shifts. 1 Introduction Vision–Language Models (VLMs) achieve strong performance by aligning visual and linguistic representations through large-scale joint pre-training ( Radford et al. 2021 ; Jia et al. 2021 ; Li et al. 2021 ; Li et al. 2022b ) . By grounding images in high-level semantic concepts, VLMs enable unified cross-modal reasoning and consistently surpass vision-only models in image classification, captioning, and visual question answering (VQA) ( Alayrac et al. 2022 ; Li et al. 2023a ; Chen et al. 2024 ; Liu et al. 2024a ; Liu et al. 2024b ) . Unlike conventional CNNs relying on local pixel statistics ( Radosavovic et al. 2020 ; Tan and Le 2021 ; Liu et al. 2022b ) , VLMs interpret images through semantic abstraction, enhancing generalization across tasks and environments. Figure 1: (a) Raindrop streaks and brightness variations disrupt correlated appearance cues, leading to a misalignment between vision and language in image captioning. (b) Vision–language reasoning is decoupled into input, representation, and semantic decision spaces. However, this semantic-centered design introduces a unique vulnerability. Semantic representations in VLMs depend on the global coherence of visual cues such as shape, color, brightness, texture, and spatial layout ( Li et al. 2022a ; Zhang 2025 ; Covert et al. 2025 ) . When these cues are systematically disturbed by changes in appearance or illumination, image–text alignment can deteriorate even if objects remain visually recognizable. Unlike pixel-level noise ( Zhao et al. 2023c ; Zhang et al. 2025 ) , such structured perturbations directly distort semantic representations and impair VLM reasoning. Real-world weather conditions such as rain, fog, and snow naturally cause correlated disruptions through interacting physical factors. As shown in Figure 1 (a), in image captioning, weather-induced perturbations jointly distort visual appearance and contextual cues, causing misalignment between images and text by corrupting the semantic grounding of the primary subject. Despite the ubiquity of such conditions in real-world environments, existing robustness evaluations of VLMs mainly target synthetic noise or simple, isolated corruptions, such as uniform brightness shifts or Gaussian noise ( Chen et al. 2023 ; Qiu et al. 2025 ; Zhao et al. 2023d ; Liu et al. 2025 ) . These settings fail to reflect the compound nature of real weather, where multiple physical factors jointly shape semantic perception. Consequently, current benchmarks likely underestimate the vulnerability of VLMs in complex environments ( Ying et al. 2024 ; Zhao et al. 2023a ) . Motivated by this gap, we study the semantic robustness of VLMs under realistic rain from an adversarial perspective. Instead of passive testing, we employ adversarial probing to reveal how complex weather disrupts image–text alignment beyond synthetic or pixel-level perturbations. Directly optimizing physical weather parameters is difficult due to the strong coupling between visual and linguistic representations in VLMs, which often yields shallow appearance changes without meaningful semantic shifts ( Zhao et al. 2023b ; Usama et al. 2025 ) . To overcome this, we propose a two-stage Semantic Decoupling framework that separates semantic manipulation from physical appearance synthesis, as illustrated in Figure 1 (b). The framework decomposes vision–language reasoning into the input image, perceptual representation, and semantic decision spaces, exposing vulnerabilities at the perceptual level before realizing them through physically grounded weather synthesis. Specifically, our framework consists of two stages. In Stage 1 , we modulate the image–text embedding space to weaken the semantic alignment learned by VLMs. This design is motivated by the observation that realistic rainy conditions primarily affect high-level semantics rather than isolated pixels. By relaxing semantic boundaries in advance, subsequent weather-induced effects can more effectively induce semantic misalignment. In Stage 2 , we optimize a parameterized rainy-weather space that jointly models multi-scale raindrops and spatial illumination. Using CMA-ES ( Nomura and Shibata 2024 ) , we generate physically plausible rainy patterns that exploit the loosened semantic structure to maximize semantic misalignment. Unlike approaches that accumulate pixel-level noise, our framework provides an interpretable and physically grounded perspective on how different weather components contribute to semantic degradation in VLMs. Compared to prior work, our method produces realistic perturbations, improves optimization stability through semantic decoupling, and demonstrates strong transferability across tasks and model architectures. Our contributions are summarized as follows: • We are the first to systematically study the robustness of VLMs under rainy conditions, with a focus on semantic degradation rather than pixel-level distortions. • We propose a two-stage framework based on semantic decoupling that aligns with the semantic reasoning mechanism of VLMs and the compound nature of weather effects. • We evaluate our approach on zero-shot image classification, image captioning, and visual question answering, and provide detailed analyses of semantic shifts. 2 Related Work Text-side Manipulation and Sensitivity Analysis. Text-side variations provide an alternative pathway for influencing vision–language model inference by manipulating linguistic inputs without modifying visual content. Changes in descriptions, prompt templates, or phrasing can substantially affect model predictions, as evidenced by the strong prompt sensitivity observed in CLIP-style zero-shot classification ( Radford et al. 2021 ) . Subsequent prompt-learning approaches introduce learnable textual prototypes or contextual vectors, further emphasizing the dominant role of language in shaping cross-modal alignment ( Zhou et al. 2022 ; Khattak et al. 2023 ) . Beyond performance effects, recent studies demonstrate that adversarially crafted prompts can steer model behavior or induce unintended and potentially unsafe outputs without altering the visual input ( Liu et al. 2022a ; Dong et al. 2023 ) . Visually Grounded Adversarial Attacks without Physical Constraints. Small-magnitude pixel-space perturbations are widely used to attack VLMs by disrupting cross-modal alignment. By transferring CNN-based adversarial techniques to CLIP-style architectures, prior studies show that visual-only perturbations can substantially degrade cross-modal matching and downstream performance ( Li et al. 2023b ; Niu et al. 2024 ) . Recent analyses ( Schlarmann and Hein 2023 ; Zhang et al. 2023 ) further reveal that such perturbations distort joint embedding

Read the complete text