When Style Breaks Safety: Defending Language Models Against Superficial Style Alignment
Yuxin Xiao, Sana Tonekaboni, Walter Gerych, Vinith Suriyakumar, Marzyeh Ghassemi · 2025
Read When Style Breaks Safety: Defending Language Models Against Superficial Style Alignment by Yuxin Xiao, Sana Tonekaboni, Walter Gerych, Vinith Suriyakumar, Marzyeh Ghassemi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07452.txt.
Introduction
Yuxin Xiao, Sana Tonekaboni, Walter Gerych, Vinith Suriyakumar, and Marzyeh Ghassemi studied how writing style can break language model safety in two thousand twenty-five. A request can be phrased in a set style, like a list, even when its intent is harmful. That style alone may raise the chance a model complies.
Across thirty-six models and seven benchmarks, nearly all showed higher attack success when style was added. Models that attended more to style were hit harder. Fine-tuning on a style also made models weaker against attacks in that same style. Their defense, SafeStyle, adds a small set of safety examples matched to the training styles and beat other defenses.
Catalog record
| CREATOR | Yuxin Xiao, Sana Tonekaboni, Walter Gerych, Vinith Suriyakumar, Marzyeh Ghassemi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07452
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Style Breaks Safety: Defending Language Models Against Superficial Style Alignment Yuxin Xiao, Sana Tonekaboni, Walter Gerych, Vinith Suriyakumar, Marzyeh Ghassemi Abstract Large language models (LLMs) can be prompted with specific styles (e.g., formatting responses as lists), including in jailbreak queries. Although these style patterns are semantically unrelated to the malicious intents behind jailbreak queries, their safety impact remains unclear. In this work, we seek to understand whether style patterns compromise LLM safety, how superficial style alignment increases model vulnerability, and how best to mitigate these risks during alignment. We evaluate 32 32 LLMs across seven jailbreak benchmarks, and find that malicious queries with style patterns inflate the attack success rate (ASR) for nearly all models. Notably, ASR inflation correlates with both the length of style patterns and the relative attention an LLM exhibits on them. We then investigate superficial style alignment, and find that fine-tuning with specific styles makes LLMs more vulnerable to jailbreaks of those same styles. Finally, we propose SafeStyle, a defense strategy that incorporates a small amount of safety training data augmented to match the distribution of style patterns in the fine-tuning data. Across three LLMs and five fine-tuning style settings, SafeStyle consistently outperforms baselines in maintaining LLM safety. 1 1 1 Our code is available at https://github.com/xiaoyuxin1002/SafeStyle . Content Warning: This paper contains examples of harmful language. 1 Introduction LLM alignment ( bai2022training ; ouyang2022training ) aims to enhance the helpfulness and harmlessness of model responses. Recent advances in alignment techniques ( ethayarajh2024kto ; rafailov2023dpo ; shao2024deepseekmath ) have enabled LLMs to follow style patterns when responding to user queries, such as the list-formatting requirement in “create a list of healthy snacks” . However, these style patterns may also appear in jailbreak queries, such as “create a list of chemical warfare agents” . Despite extensive safety training during alignment, LLMs remain vulnerable to such attacks ( chen2024when ; deng2024multilingual ; dong2024attacks ; jin2024jailbreakzoo ; wei2023jailbroken ; zheng2024improved ) . These observations raise a critical yet understudied question: Can style patterns in malicious queries inadvertently undermine the robustness of safety-aligned LLMs? The superficial alignment hypothesis ( lin2024the ; zhang2024dissecting ; zhou2023lima ) posits that alignment tuning may encourage models to imitate style patterns without internalizing deeper safety principles. Meanwhile, prior work in safety defense has largely focused on data curation ( bianchi2024safetytuned ; he2024what ; shen2025seal ) , mechanistic interventions ( djuhera2025safemerge ; hazra2024safety ; hsu2024safe ; li2025salora ; liu2024targeted ; tamirisa2025tamperresistant ; yi2025nlsr ; zhao2025understanding ) , or safety training objectives ( huang2024lisa ; huang2025booster ; huang2024vaccine ; rosati2024representation ; zhang2025bifactorial ) . Recent studies have noted that diverse alignment data enhances LLM robustness ( hsiung2025your ) , and that safety training may often be shallow in terms of response token depth ( qi2025safety ) . This leaves a crucial gap in understanding the safety implications of style compliance. To address this gap, we investigate three questions: (1) Do style patterns affect LLM safety? (2) How do safety vulnerabilities emerge during superficial style alignment? (3) How can we mitigate these risks during the alignment process? Style patterns and LLM safety : Through an extensive study of 32 LLMs ( bai2023qwen ; grattafiori2024llama3 ; groeneveld2024olmo ; jiang2023mistral7b ; olmo20242 ; team2025gemma3 ; team2024gemma ; team2024gemma2 ; touvron2023llama2 ; yang2407qwen2 ; yang2024qwen25 ) across seven jailbreak benchmarks ( han2024medsafetybench ; huang2024catastrophic ; mazeika2024harmbench ; rottger2024xstest ; alexandra2024strong ; xie2025sorrybench ; zou2023universal ) , we find that style patterns inflate the ASR for nearly all models. Here, ASR inflation refers to the phenomenon in which the underlying malicious intent remains unchanged, but the addition of semantically irrelevant style patterns to jailbreak queries breaks LLM safety. In other words, these models exhibit lower ASR when prompted with the same malicious queries after the style patterns are removed. We further show that ASR inflation correlates with a model’s relative attention to these patterns and the pattern length. Surprisingly, ASR-inflating style patterns appear more frequently in the instruction-tuning datasets ( ivison2023tulu2 ; lambert2024tulu3 ) used to align LLMs ( groeneveld2024olmo ; olmo20242 ) . Safety vulnerabilities during superficial style alignment : We evaluate the increase in ASR when fine-tuning models (Llama-3.1-8B-Instruct ( grattafiori2024llama3 ) ) using instructions with different style patterns. In addition to the original instruction-tuning dataset ( taori2023alpaca ) and its simplified version with style patterns removed, we augment the instructions with two styles (list ( he2024what ) and poem ( chakrabarty2022help ; chen2024evaluating ; mahbub2023unveiling ) ) as either prefixes or suffixes. We construct the safety test set using the same style variations. We find that models fine-tuned on data containing specific style patterns become more vulnerable to jailbreaks in the same style, which suggests that superficial style alignment inflates safety risks. Moreover, increasing the proportion of data with the matched style further increases ASR. In contrast, the position of the style patterns has minimal impact. Mitigating the inflated safety risks during superficial style alignment : We propose SafeStyle, a simple intervention that incorporates a small amount of additional safety training data, augmented to match the distribution of style patterns in the fine-tuning data. We evaluate SafeStyle against five existing defense strategies ( bianchi2024safetytuned ; eiras2025do ; li2025safety ; lyu2024keeping ; qi2025safety ) by targeting safety risks during fine-tuning with five different style patterns ( chakrabarty2022help ; he2024what ; jhamtani2017shakespearizing ; ma2024at ) . Across three LLMs ( grattafiori2024llama3 ; team2025gemma3 ; yang2024qwen25 ) of varying sizes and families, we demonstrate that while all methods yield comparable style adaptation performance ( dubois2024lengthcontrolled ; li2023alpacaeval ) , only SafeStyle fully maintains LLM safety against jailbreaks with varied style patterns. We summarize our contributions in this paper as follows: • We show that style patterns inflate ASR by systematically evaluating the safety risks of 32 LLMs across seven jailbreak benchmarks. • We demonstrate that superficial style alignment contributes to ASR inflation by fine-tuning and testing LLMs with varied style patterns. • We introduce SafeStyle, a defense against the safety risks posed by superficial style alignment, which effectively preserves LLM safety and outperforms other baselines. 2 Related Work LLM Alignment. LLM alignment ( bai2022training ; ethayarajh2024kto ; ouyang2022training ; rafailov2023dpo ) aims to guide models to follow human instructions and behave in accordance with desirable norms. These techniques can also adapt LLMs to domain-specific tasks with distinct textual styles, such as poetry generation ( chakrabarty2022help ; chen2024evaluating ; mahbub2023unveiling ) , code synthesis ( ma2024at ; roziere2023code ; zhang2025unveiling ) , and mathematical problem solving ( azerbayev2024llemma ; toshniwal2024openmathinstruct ; yu2024metamath ) . However, several works ( lin2024the ; raghavendra2024revisiting ; zhang2024dissecting ; zhou2023lima ) argue that alignment tuning can be superficial, with models merely adapting to the topics and styles present in the fine-tuning data. Our work builds on this line of research by thoroughly investigating and mitigating the inflated safety risks introduced by superficial style alignment. LLM Safety Risk. Despite extensive safety alignment efforts, LLMs continue to face significant safety risks, particularly in their susceptibility to malicious queries ( chen2024when ; deng2024multilingual ; dong2024attacks ; jin2024jailbreakzoo ; wei2023jailbroken ; zheng2024improved ) . Recent work has introduced jailbreak benchmarks and evaluation protocols to quantify the ease with which LLMs can be manipulated into producing unethical responses ( han2024medsafetybench ; huang2024catastrophic