Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models

Hashmat Shadab Malik, Muzammal Naseer, Salman Khan · 2026

Read Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models by Hashmat Shadab Malik, Muzammal Naseer, Salman Khan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03793.txt.

Introduction

Malik, Naseer, and Khan map adversarial weakness in multimodal models across twelve languages. Vision inputs give such models a continuous attack surface. Past robustness work centered on English tasks. Multilingual behavior stayed unexplored until now. Open-source models that gain languages through instruction tuning are evaluated.

Gradient attacks transfer across languages with ease. An image tuned in one language still breaks the model in others. Safety varies with how well a model reads harmful intent. Strong languages trigger misuse more often through text. Harmful text inside images is followed in English but rarely parsed in other scripts. Low-resource languages can thus look safer by failing to understand, a trap named safety-by-failure. Models trained multilingually throughout, like Qwen3-VL, refuse actively across languages instead. Shallow fine-tuning on translated data yields only surface safety. Deep integration yields the real thing.

Catalog record

CREATORHashmat Shadab Malik, Muzammal Naseer, Salman Khan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03793

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models Hashmat Shadab Malik, Muzammal Naseer, Salman Khan Abstract Multimodal Large Language Models (MLLMs) integrate visual perception into language reasoning, but in doing so they introduce a continuous attack surface that is susceptible to adversarial attacks. Although prior works have examined their robustness, most evaluations are restricted to English-centric tasks, leaving behaviour in multilingual settings unexplored. We address this gap through a systematic study of adversarial robustness and multimodal safety across 12 typologically diverse languages. We primarily evaluate representative open-source MLLMs that acquire multilingual capability through multimodal instruction tuning. Our results on gradient-based adversarial attacks reveal a transferable multilingual vulnerability, whereby adversarial images optimized in one language continue to induce failure when evaluated in others, demonstrating strong cross-lingual transferability . Our safety analysis further shows that multilingual safety behaviour varies across languages depending on how effectively a model can retrieve or interpret harmful instructions. When harmful intent is issued through the textual modality, languages with stronger linguistic grounding are more likely to elicit misuse-enabling responses, while weaker languages produce far fewer unsafe outputs. Similarly, when the harmful query is embedded in the vision modality as typographic content, English scripts are reliably recognised and followed, whereas non-English scripts are rarely parsed by the vision encoder. Lower-resource languages and non-English scripts may therefore appear safer in both channels, but this apparent robustness is an artefact of comprehension and visual-grounding failures rather than genuine safety alignment, a phenomenon we term safety-by-failure . In contrast, MLLMs that build multilingual capability throughout their training stages rather than only at instruction tuning, such as Qwen3-VL , exhibit genuine cross-lingual safety, maintaining active refusal across languages rather than masking comprehension failure. Together, these findings highlight that shallow multilingual adaptation, such as fine-tuning just on translated instruction data, may produce surface-level multilingual understanding that creates illusory safety in low-resource languages; a deeper integration of multilingual capability across training stages leads to more genuine multilingual understanding and safety alignment. Our code and benchmark adapted for the multilingual setting will be made publicly available. Adversarial Robustness and Safety Alignment in MLLMs † † email: [email protected] † † email: [email protected] † † email: [email protected] † † affiliation: Mohamed Bin Zayed University of AI, UAE † † affiliation: Khalifa University, UAE † † affiliation: Australian National University, Australia 1 Introduction Large Language Models (LLMs) [ 34 , 9 , 30 , 1 , 31 ] have demonstrated remarkable capabilities in reasoning, knowledge representation, and instruction following. Building on this foundation, Multimodal Large Language Models (MLLMs) extend these capabilities to the visual domain by integrating pretrained vision encoders [ 35 ] with LLM backbones via lightweight projection modules [ 25 , 21 ] . This modular architecture allows MLLMs to inherit the linguistic capabilities of the LLM backbone while grounding reasoning in visual content through the vision encoder. As MLLMs are increasingly deployed in global settings, multilingual capability has emerged as a critical requirement. Ideally, MLLMs should understand and respond in the same language as the user input, ensuring equitable access across regions and cultures [ 11 , 17 ] . However, a significant performance gap exists: while state-of-the-art foundation LLMs exhibit impressive cross-lingual competence [ 16 , 45 , 1 ] , this capability is often compromised during the vision–language alignment phase. In practice, leading models like LLavA [ 25 ] exhibit multilingual erosion [ 2 ] ; a phenomenon where models default to English or suffer from degraded reasoning and instruction fidelity when handling non-English inputs. Recent efforts to mitigate this typically apply multilingual multimodal instruction tuning on top of LLavA -style architectures, for instance through translated multimodal instruction data [ 27 ] or with architectural inductive biases such as text-guided mixture-of-experts adapters for cross-lingual representation alignment [ 40 ] . More recently, model families such as Qwen3-VL [ 7 ] take a fundamentally different approach: building on a strongly multilingual LLM backbone, they incorporate high-quality multilingual multimodal data across multiple stages of MLLM training rather than relying on translated English-based instruction data. Parallel to these efforts, recent studies have identified significant adversarial and safety vulnerabilities in MLLMs [ 37 ] . Unlike discrete text tokens, visual inputs reside in a continuous, high-dimensional space that is particularly amenable to gradient-based optimization [ 10 ] , making MLLMs vulnerable to adversarial image perturbations that can substantially degrade performance or manipulate model behavior [ 33 ] . Beyond imperceptible adversarial noise, MLLMs are also susceptible to non-adversarial visual manipulations [ 26 , 22 ] , such as visual jailbreaks, where semantically meaningful images induce harmful, disallowed, or policy-violating outputs despite a safety-aligned language backbone. Despite the gravity of these risks, evaluations of multimodal robustness have remained overwhelmingly English-centric, relying on English-language prompts and benchmarks. Consequently, a critical gap persists: it is unclear how vulnerable these models are across other languages. In this work, we bridge this gap by conducting a systematic study of multilingual adversarial robustness and multimodal safety in MLLMs. We primarily evaluate two representative open-source LLavA -based multilingual MLLMs, Palo [ 27 ] and Parrot [ 40 ] , both of which acquire multilingual capability via multilingual multimodal instruction tuning; the dominant open-source multilingual adaptation paradigm. While these models report multilingual capability across a range of benchmarks, we ask whether multilingual multimodal instruction tuning alone yields multilingual capability that is also robust to adversarial perturbations and consistently safety-aligned across languages. To examine whether models with more deeply integrated multilingual training exhibit qualitatively different safety behaviour, we additionally evaluate Qwen3-VL [ 7 ] on our non-adversarial safety benchmark. Since our gradient-based adversarial study requires white-box access to model parameters, it is restricted to the fully open-source Palo and Parrot ; extending multilingual adversarial evaluation to proprietary or API-only systems (e.g., via transfer-based or query-based attacks) remains an important open direction. To enable this evaluation, we construct a comprehensive multilingual benchmark suite spanning 12 typologically diverse languages. Since existing adversarial robustness and multimodal safety benchmarks are exclusively English-centric, we systematically adapt COCO [ 24 ] , Flickr30k [ 32 ] , LLaVA-Bench [ 25 ] , RealToxicityPrompts [ 14 ] , and MM-SafetyBench [ 26 ] into 12 languages through a rigorous translate–then–verify protocol. Translations are generated by a pool of multilingual LLMs, filtered via automated back-translation consistency checks, and validated by native or proficient speakers to ensure semantic fidelity and linguistic naturalness. For typographic safety benchmarks [ 26 ] that render harmful text directly inside images, we additionally perform manual visual inspection to verify correct typography rendering. The resulting multilingual evaluation suite, comprising over 60,000 adapted instances, provides a reusable, publicly available resource for future work on multilingual multimodal robustness and safety. We assess these models across 12 typologically diverse languages along two complementary evaluation axes: (i) robustness of multilingual captioning and reasoning under gradient-optimized visual perturbations, with a focus on cross-lingual adversarial transferability (evaluated on Palo and Parrot ), and (ii) safety behaviour under non-adversarial multimodal inputs, including textual and visual jailbreak benchmarks [ 26 , 33 ] (evaluated on Palo , Parrot , and Qwen3-VL ). Our key findings are summarized below: • Cross-Lingual Transferability. We find that MLLMs are consistently vulnerable to gradient-based adversarial perturbations across captioning and

Read the complete text