Steering Multimodal Large Language Models Decoding for Context-Aware Safety
Zheyuan Liu Zhangchen Xu Guangyao Dou Xiangchi Yuan, Zhaoxuan Tan Radha Poovendran Meng Jiang · 2025
Read Steering Multimodal Large Language Models Decoding for Context-Aware Safety by Zheyuan Liu Zhangchen Xu Guangyao Dou Xiangchi Yuan, Zhaoxuan Tan Radha Poovendran Meng Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19212.txt.
Introduction
Multimodal language models that see images as well as text often misjudge safety: they refuse harmless requests or miss risks that are plain in the picture. Zheyuan Liu and six co-authors introduce SafeCoDe, a decoding method that steers generation toward context-aware safety without retraining the model. It works with any model of this kind and adds little weight. The first stage contrasts the real image with a blurred, noised version to spotlight tokens that depend on visual context.
The second stage blends scene-level reasoning with token-level adjustment, shaping refusals to match the predicted safety verdict. Across several model families and safety benchmarks, SafeCoDe improved refusal behavior where refusal was warranted and reduced needless refusals. Helpfulness on ordinary tasks was preserved. The authors present the method as a practical patch for a persistent gap in multimodal safety alignment.
Catalog record
| CREATOR | Zheyuan Liu Zhangchen Xu Guangyao Dou Xiangchi Yuan, Zhaoxuan Tan Radha Poovendran Meng Jiang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-19212
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Steering Multimodal Large Language Models Decoding for Context-Aware Safety Zheyuan Liu Zhangchen Xu Guangyao Dou Xiangchi Yuan, Zhaoxuan Tan Radha Poovendran Meng Jiang Abstract Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisions remains limited. Existing methods often fail to balance oversensitivity (unjustified refusals of benign queries) and undersensitivity (missed detection of visually grounded risks), leaving a persistent gap in safety alignment. To address this issue, we introduce Safety-aware Contrastive Decoding (SafeCoDe), a lightweight and model-agnostic decoding framework that dynamically adjusts token generation based on multimodal context. SafeCoDe operates in two stages: (1) a contrastive decoding mechanism that highlights tokens sensitive to visual context by contrasting real and Gaussian-noised images, and (2) a global-aware token modulation strategy that integrates scene-level reasoning with token-level adjustment to adapt refusals according to the predicted safety verdict. Extensive experiments across diverse MLLM architectures and safety benchmarks, covering undersensitivity, oversensitivity, and general safety evaluations, show that SafeCoDe consistently improves context-sensitive refusal behaviors while preserving model helpfulness. 1 1 1 Code is available at franciscoliu/SafeCoDe . Figure 1: Illustration of multimodal situational safety. The model must judge the safety of a user’s query based on the accompanying visual context and adjust its response accordingly. In unsafe contexts (undersensitivity), the model should issue a refusal or warning rather than directly answering the harmful request. In benign contexts (oversensitivity), the model should avoid unnecessary refusals and provide a helpful response. However, current MLLMs struggle with both types of errors. 1 Introduction The rapid advancement of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has led to significant breakthroughs across a broad range of AI tasks ( Liu et al. 2024a ; Chowdhery et al. 2023 ; Liu et al. 2024f ; Zhu et al. 2023 ; Zhang et al. 2025b ; Ni et al. 2025 ; Zhang et al. 2025a ; Qin et al. 2023 ; Ouyang et al. 2022 ; Tan et al. 2024 ) . These models demonstrate remarkable capabilities in understanding complex multimodal contexts, following nuanced instructions, and generating high-quality human-readable outputs. With their growing deployment in real-world applications, ensuring the safety of MLLMs has become a pressing concern, emerging a number of research works dedicated to mitigating harmful, unethical, or misleading behaviors ( Li et al. 2024a ; Gong et al. 2025 ; Zhou et al. 2024 ; Shayegani et al. 2023 ) . Current MLLM safety evaluations ( Liu et al. 2024d ; Qi et al. 2024 ; Liu et al. 2024c ) predominantly treat textual queries as potential attack vectors, with visual inputs serving as auxiliary attackers. However, as MLLMs become increasingly capable of integrating vision and language, the visual context often plays a pivotal role in determining the appropriateness of a response. This shift introduces the challenge of undersensitivity ( Zhou et al. 2024 ) , where a model fails to recognize visually implied harm in otherwise benign-looking queries. For instance, as shown on the left side of Figure 1 , although the user’s question appears harmless in isolation (e.g., practicing a swing), the accompanying image reveals a hazardous office setting, warranting a safety-aware refusal. In contrast, recent findings ( Li et al. 2024a ) have also highlighted the importance of oversensitivity in MLLMs, wherein the model refuses to answer benign queries due to overly cautious safety triggers. As illustrated on the right of Figure 1 , a user who asks a simple question in a visually normal context (e.g., describing an ordinary parade) receives an unnecessary refusal due to the model’s misinterpretation of potential violence. These contradictory failure modes, under and ovesensitivity, highlight a critical limitation in current safety alignment techniques, which often operate under static assumptions about harmfulness. This duality raises an urgent research question: How can we enable MLLMs to make context-aware safety decisions that avoid undersensitivity to risks and oversensitivity to benign queries? Addressing this requires models to make appropriate safety decisions that are multimodal and situationally grounded. In this work, we begin by systematically evaluating state-of-the-art safety alignment techniques for MLLMs. Our findings reveal a critical imbalance: existing methods are often undersensitive (overlooking unsafe inputs) or oversensitive (rejecting benign queries). As a result, they struggle to achieve robust performance across both context-sensitive and general safety benchmarks. To address this gap, we introduce Safe ty-aware Co ntrastive De coding (SafeCoDe), a contrastive, context-aware safety decoding framework that adaptively modulates token generation based on fine-grained differences in visual context and intent cues. In particular, SafeCoDe enhances contextual safety alignment by dynamically adjusting token probabilities, suppressing unsafe completions in risky contexts while preserving helpfulness in benign queries. Through a two-stage design with contrastive signal initialization, which reduces oversensitivity by grounding refusals in visual information, and global-aware token modulation, which mitigates undersensitivity by leveraging scene-level reasoning to capture subtle risks, SafeCoDe enables models to achieve robust safety alignment across both under and oversensitivity regimes. To sum up, our contributions are listed as follows: 1. We investigate the intricate balance of contextual safety in MLLMs and highlight key limitations of prior alignment methods. While existing approaches show strong performance in general safety and jailbreak settings, they fall short in incorporating holistic multimodal context, making it difficult to balance oversensitivity and undersensitivity by refusing harmful queries reliably while remaining helpful on benign ones. 2. We propose SafeCoDe, a novel real-time decoding framework that dynamically integrates both visual and textual cues to modulate early-stage token generation. Through a two-stage design, SafeCoDe enables fine-grained safety control conditioned on multimodal context. 3. Finally, we conduct extensive experiments and case studies to demonstrate the effectiveness of SafeCoDe in achieving context-sensitive safety alignment. Our results show that SafeCoDe consistently reduces both over and undersensitivity across a range of safety-critical scenarios while preserving the model’s utility on general-purpose multimodal tasks. 1.1 Key Observations and Insights Over-reliance on Textual Modality. Our first motivation comes from recent evidence ( Leng et al. 2024 ) that MLLMs often exhibit strong unimodal bias, relying heavily on textual priors while underutilizing visual inputs. We ask whether a similar issue also undermines contextual safety. To test this, we replace all images in contextual safety benchmarks with blank placeholders, representing a lack of image modality information. Then we compare the model performance with the original image condition. As shown in Table 1 , performance remains nearly unchanged across both settings, revealing that base MLLMs anchor their refusals primarily on textual input while neglecting visual context. This behavior poses a risk for safety alignment, as refusals are issued (or withheld) based on language priors rather than situational evidence in the scene. To further validate this observation, we examine a variant with contrastive decoding enabled. Unlike the base model, this setting shows clear gains, demonstrating that explicitly contrasting real and neutralized images encourages the model to rely on visual grounding rather than text-only heuristics. Figure 2: Undersensitivity (top): Accuracy on unsafe cases in MSSBench, where higher values indicate stronger ability to block harmful queries. Oversensitivity (bottom): Rejection rates on MOSSBench, where lower values indicate fewer unnecessary refusals of benign queries. Results are shown for multiple baselines and our method SafeCoDe. Models MSSBench (Accuracy) ( \uparrow ) MOSSBench (Rejection Rate) ( \downarrow ) Safe (Chat) Unsafe