Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs
Casey Ford, Madison Van Doren, Sicheng Jin, Emily Dix · 2026
Read Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs by Casey Ford, Madison Van Doren, Sicheng Jin, Emily Dix (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23157.txt.
Introduction
Ford and three coauthors run the systematic cross-lingual multimodal red teaming study of jailbreak vulnerability in English and Spanish across four frontier models. Using 363 scenarios in text and multimodal conditions, with 52,272 harm ratings from native speaker panels, they find language does not scale vulnerability uniformly. Role play attacks weaken under Spanish prompting, while multimodal attacks strengthen. This implicates the language interface rather than annotator leniency. Safety rankings flip across languages. Qwen Omni passes Pixtral Large as vulnerable in Spanish, a reversal no correction of English scores could recover.
Catalog record
| CREATOR | Casey Ford, Madison Van Doren, Sicheng Jin, Emily Dix |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60715 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs Casey Ford, Madison Van Doren, Sicheng Jin, Emily Dix Abstract The attack surface of a multimodal large language model (MLLM) is language-dependent in ways that reveal the mechanistic structure of alignment failures. We present the first systematic cross-lingual, multimodal red-teaming study comparing jailbreak vulnerability in US English (en-US) and Mexican Spanish (es-MX) across four frontier MLLMs: Claude Sonnet 4.5, GPT-5, Pixtral Large, and Qwen Omni. Using a fixed adversarial benchmark of 363 diverse prompt scenarios administered in text-only and multimodal conditions, we collected 52,272 harm ratings and binary attack success judgements from matched panels of nine native-speaker annotators per language group. Our central finding is that language does not scale vulnerability uniformly. Bayesian mixed-effects analyses reveal that linguistic framing attacks such as role-play become substantially less effective under Spanish prompting, while visually explicit multimodal attacks become more effective, which directly implicates the prompt-language interface rather than global annotator leniency. This dissociation indicates that linguistic and visual alignment failures operate through distinct mechanisms, and that switching language is sufficient to expose that separation. The practical consequence is that safety rankings are not preserved across languages. Qwen Omni overtakes Pixtral Large as the most vulnerable model among es-MX participants, a rank reversal no scalar correction of English-condition scores could recover, and absolute attack success rates have declined across model generations without closing the gaps between them. These findings demonstrate that safety evaluation frameworks treating language and modality as independent dimensions fundamentally misspecify the attack surface of globally deployed MLLMs, and must be redesigned accordingly. 1 Introduction The attack surface of a multimodal large language model (MLLM) is not a fixed property of the model. It is structurally language-dependent in ways that reveal how alignment actually fails. We demonstrate this by showing that switching adversarial prompt language from US English (en-US) to Mexican Spanish (es-MX) is sufficient to expose a dissociation: linguistic attack techniques that succeed in English lose effectiveness in Spanish, while visually explicit multimodal attacks gain effectiveness. This pattern is mechanistically interpretable as evidence that linguistic and visual alignment failures operate through distinct pathways, and that alignment training as currently practised embeds language-specific sensitivity rather than language-agnostic safety. The explanation connects directly to what is already known about how alignment training works. RLHF-based alignment learns from human feedback on English-language outputs, embedding model sensitivity to the rhetorical patterns through which jailbreaks operate in English: role-play framing, strategic persuasion, refusal suppression. This language-conditioned calibration means that prompts exploiting these patterns in Spanish encounter a model tuned to recognise them in English, but not in Spanish. Visual attack channels, by contrast, engage processing pathways that are less tied to natural language: the relationship between an image and its adversarial intent is not processed through the same linguistically-calibrated mechanisms as rhetorical framing. If the account is correct, the dissociation we observe is the predicted outcome of the interaction between alignment mechanisms and language, not a post-hoc surprise. Van Doren & Ford 38 established the empirical baseline for this inquiry: across a matched set of red-teaming scenarios administered to four MLLMs in English, they found that model susceptibility varied substantially across modalities in model-specific ways, and identified cross-lingual evaluation as a priority for future work. The present study directly addresses that priority. Using the same validated adversarial benchmark of 363 unique prompts, each administered in text-only and multimodal conditions, we evaluate four next-generation successors of the models studied in that prior work: Claude Sonnet 4.5, GPT-5, Pixtral Large, and Qwen Omni. Prompts were professionally translated into Mexican Spanish and submitted to each model independently in both languages; nine trained en-US annotators rated English-language model responses, and nine trained es-MX annotators rated Spanish-language model responses, yielding 52,272 observations modelled using Bayesian mixed-effects analyses with crossed random effects of prompts and raters. Our contributions are threefold: (1) We provide the first mechanistic evidence, from a matched bilingual red-teaming design with human-translated prompts and native-speaker annotation, that linguistic and visual alignment failures operate through distinct pathways. The dissociation between linguistic attack attenuation and visual attack amplification under Spanish prompting directly implicates the prompt-language interface as a structural feature of alignment, not a nuisance artefact of annotator leniency. We additionally show that relative safety rankings are largely preserved across the model generations studied here, while absolute attack success rates have declined in three of the four model families. (2) We show that the practical consequence of language-dependent alignment is that safety rankings are not transferable across the en-US/es-MX contrast: Qwen Omni overtakes Pixtral Large as the most vulnerable model among es-MX participants, which is a rank reversal no scalar correction of English-condition scores could recover. This extends existing evidence on mixed-language vulnerability 12 and establishes that cross-lingual evaluation is a structural problem for safety assessment rather than one that scales away with model improvement. (3) We demonstrate that a Bayesian mixed-effects framework, by jointly accounting for prompt-level and rater-level variance, reveals cross-lingual and cross-modal structure that aggregate analysis would miss, and provide reusable analysis scripts to facilitate adoption of this approach in future safety evaluations. 2 Related Works The safety of LLMs has become a central concern, with adversarial prompting established as a key method for stress-testing vulnerabilities 14 ; 36 ; 41 . Research has shown that subtle manipulations can bypass safeguards to produce potentially harmful outputs 18 ; 20 ; 28 , and surveys have catalogued the breadth of such vulnerabilities 33 ; 34 . More recent work has emphasised that apparent safety outcomes can be influenced by refusal behaviours rather than robust alignment 47 , and that mechanisms relying on explicit refusal prefixes are themselves vulnerable to prefix-injection attacks 43 . Multi-step and multi-turn attack frameworks, including task concurrency 21 , iterative persuasion 42 , and reasoning-model exploitation 17 , expose failure modes that single-turn benchmarks systematically miss. New benchmarks demonstrate persistent vulnerabilities across SOTA models despite iterative safety improvements 48 ; 4 . The key mechanistic insight motivating the present study is that alignment is language-conditioned. Deng et al. 10 demonstrated that multilingual safety failures are embedded in the pre-training stage and do not improve with RLHF: the cross-lingual alignment bottleneck is not a fine-tuning problem. Shen et al. 35 confirmed that optimising for popular languages cannot produce language-agnostic safety, and Yong et al. 46 showed that converting simple malicious English prompts to under-resourced languages causes attack success rates to rise dramatically. Emani et al. 12 further showed that models remain susceptible to malicious instructions in mixed-language contexts. Together, these results indicate that the RLHF-based alignment embeds sensitivity to the specific rhetorical patterns through which jailbreaks operate in English, making that sensitivity unavailable when the same attack strategies are expressed in other languages. Figure 1 : Elicitation of adversarial prompt dataset A parallel body of work establishes that visual attack channels operate through mechanisms that are less language-conditioned. Images are a persistent alignment weak point: vision-language jailbreaks successfully bypass safeguards 27 ; 29 , universal multimodal attacks generalise across model families despite alignment strategies 40 , and distraction-based constructions can systematically overload safety mechanisms in closed-source MLLMs 45 . Audio represents a further modality-specific attack surface operating through distinct