Where Do Prompt Perturbations Break Generation? A Segment-Level View of Robustness in LoRA-Tuned Language Models
Zhuoyun Li, Boxuan Wang, Jinwei Hu, Zhenglin Huang, Qisong He · 2026
Read Where Do Prompt Perturbations Break Generation? A Segment-Level View of Robustness in LoRA-Tuned Language Models by Zhuoyun Li, Boxuan Wang, Jinwei Hu, Zhenglin Huang, Qisong He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01605.txt.
Introduction
Zhuoyun Li and eight co-authors locate exactly where a small prompt typo breaks a language model's answer. Robustness work usually grades the whole response, which can hide a drift that hits one key entity, relation, or conclusion while the rest reads fine.
Their framework works at segment level. Clean and perturbed answers are cut into meaning segments, aligned by an optimal-transport objective, and the worst-drifting segments are penalized. An adapter-stability term, guided by how attention reallocates across segments, keeps the low-rank tuning from amplifying those shifts; a PAC-Bayesian view explains why capping adapter growth should transfer to unseen noise. On summarization tests the method hardens models against typos, deletions, synonym swaps, and paraphrases, keeps clean scores competitive, and transfers across data sets better than whole-sequence rivals.
Catalog record
| CREATOR | Zhuoyun Li, Boxuan Wang, Jinwei Hu, Zhenglin Huang, Qisong He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01605
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Where Do Prompt Perturbations Break Generation? A Segment-Level View of Robustness in LoRA-Tuned Language Models Zhuoyun Li, Boxuan Wang, Jinwei Hu, Zhenglin Huang, Qisong He Abstract Large language models are sensitive to minor prompt perturbations, yet existing robustness methods usually enforce consistency at the whole-sequence level. This holistic view can hide an important failure mode: a perturbed response may remain globally similar to the clean one while drifting on a critical entity, relation, or conclusion. We introduce S 2 R 2 , a segment-level framework for robust LoRA fine-tuning. S 2 R 2 decomposes clean and perturbed generations into semantic segments, aligns them with an optimal-transport objective, and penalises the segments with the largest meaning drift. To connect this output-side objective with model adaptation, we add an adapter-stability regulariser motivated by segment-level attention reallocation, using LoRA norm control as a tractable proxy for limiting perturbation-amplified evidence shifts. A PAC-Bayesian complexity view further explains why controlling adapter growth may support transfer beyond observed perturbations. Experiments on summarisation benchmarks show that S 2 R 2 improves robustness under typographical noise, deletion, synonym replacement, and paraphrasing, while maintaining competitive clean performance and stronger cross-dataset transfer than consistency-based baselines. 1 Introduction Large Language Models (LLMs) have achieved widespread adoption in numerous applications ( 13 ; 28 ; 11 ) , but their reliability is often compromised by minor perturbations to input prompts ( 15 ; 29 ) . These perturbations can arise from typographical errors, paraphrases, word-level substitutions, or adversarially crafted changes, and may lead to unreliable or unsafe outputs ( 34 ; 25 ; 22 ; 19 ; 7 ; 24 ; 20 ; 30 ) . For generation tasks such as summarisation, even a small change in a key entity, relation, or conclusion can alter the meaning of the entire response. This fragility undermines the trustworthy deployment of LLMs, especially in domains where output faithfulness and semantic stability are critical. Many researchers have focused on bolstering LLM robustness and have proposed well-designed fine-tuning strategies ( 23 ; 33 ; 2 ; 37 ; 16 ) . A common thread in these methods is a holistic treatment of the output, for example by minimising the Kullback-Leibler (KL) divergence or related consistency losses over an entire sequence. While such objectives are effective for improving average robustness, they disregard a basic property of language: semantic information is unevenly distributed across an output. Just as a few keywords can define the meaning of a sentence, particular segments of an LLM output may be more critical to its semantic integrity than others. This non-uniform vulnerability is also consistent with observations in robust learning and adversarial fairness, where different groups or components can contribute unevenly to the overall risk ( 1 ; 9 ; 17 ) . By ignoring this asymmetry, holistic methods may hide local failures in which the overall response remains similar while a critical semantic segment drifts ( 23 ) . This motivates a segment-level view of prompt robustness. Instead of asking only whether the full clean and perturbed outputs are similar, we ask where the generation breaks. This distinction is important because perturbations often do not damage all parts of the output uniformly. A response may preserve most of its surface form while changing a medical condition, numerical value, causal relation, or final conclusion. In such cases, sequence-level consistency can underestimate the true robustness failure. A robust fine-tuning objective should therefore identify and stabilise the semantic segments where perturbations induce the largest meaning shift. Focusing on output-side semantic drift alone, however, only captures part of the picture. Transformer generation is mediated by attention mechanisms ( 26 ) , and input perturbations can affect both token representations and the way contextual evidence is allocated during generation ( 7 ; 3 ) . We do not claim that attention provides a complete causal explanation of model behaviour. Instead, we use segment-level attention reallocation as a tractable diagnostic proxy: it describes how the model redistributes attention mass across source or evidence segments when the input is perturbed. This view complements the output-side semantic loss by offering a mechanism-side interpretation of why certain output segments become unstable. In this work, we introduce Semantic Segment Robustness Regularisation (S 2 R 2 ), a robust fine-tuning framework for LoRA-adapted LLMs ( 10 ) . S 2 R 2 first decomposes clean and perturbed generations into semantic segments, aligns them through an optimal-transport objective, and penalises the segments with the largest semantic drift. It then adds an adapter-stability regulariser that controls the growth of LoRA updates in attention projections. This term should not be interpreted as a standalone new norm penalty. Rather, it serves as a tractable proxy for limiting perturbation-amplified attention reallocation under LoRA adaptation. Since LoRA updates a frozen pre-trained model through low-rank matrices, it also provides a convenient adaptation space for analysing model movement and complexity ( 10 ) . We further connect this adapter-stability term to a PAC-Bayesian complexity view ( 21 ) . Under a standard interpretation where LoRA parameters define a posterior around the frozen pre-trained model, controlling adapter growth can be understood as limiting the complexity of the adapted hypothesis. We use this as a generalisation-oriented explanation rather than as a certification claim. In other words, S 2 R 2 is designed to improve robustness by preserving worst-drift semantic segments while discouraging excessive adapter movement away from the pre-trained model. Fig. 1 illustrates the overall framework. A clean input and its perturbed variant are processed by a LoRA-tuned LLM. The output-side objective identifies and penalises segment-level semantic drift, while the mechanism-side stability term constrains the LoRA updates that may amplify perturbation-induced evidence reallocation. Together with the standard cross-entropy objective, these components encourage the model to preserve critical semantic content under prompt perturbations. Figure 1: Overview of S 2 R 2 . The framework aligns clean and perturbed generations at the segment level and emphasises the worst-drift semantic segments. The mechanism-side branch, labelled as attention shift in the schematic, is implemented as an adapter-stability term L_{\mathrm{stab}} that controls LoRA update growth as a proxy for limiting perturbation-amplified attention reallocation. Together with L_{\mathrm{CE}} and L_{\mathrm{sem}} , this encourages robust generation by preserving critical semantic content rather than only enforcing whole-sequence consistency. To summarise, the main contributions of our paper are as follows: C1: We propose a segment-level view of prompt robustness, arguing that robustness failures in generation should be analysed through local semantic drift rather than only through whole-sequence consistency. C2: We introduce S 2 R 2 , a robust LoRA fine-tuning framework that aligns clean and perturbed generations at the semantic-segment level and explicitly penalises the worst-drift segments. C3: We connect segment-level robustness with adapter stability by interpreting LoRA norm control as a tractable proxy for limiting perturbation-amplified attention reallocation and as a PAC-Bayesian complexity-oriented regulariser ( 21 ) . 1 1 1 For transparency, we note that an LLM was used to assist with language polishing. See detailed statement in App. D . The source code for this paper will be made publicly available upon acceptance. C4: We empirically evaluate S 2 R 2 on summarisation benchmarks, showing improved robustness under multiple prompt perturbations and stronger cross-dataset transfer than consistency-based baselines. 2 Preliminaries To