Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models

Runpeng Dai, Run Yang, Fan Zhou, Hongtu Zhu · 2025

Read Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models by Runpeng Dai, Run Yang, Fan Zhou, Hongtu Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03714.txt.

Introduction

Runpeng Dai, Run Yang, Fan Zhou, and Hongtu Zhu probe why large language and vision models can be fooled by small changes. Their paper is from two thousand twenty-five. A tiny tweak to an input or a weight can flip a strong model's answer. The sources of that weakness were not pinned down.

They build a score called FI, for first-order local influence, based on information geometry. It gauges how much each weight or input part sways the output. Across models of many sizes, a small set of high-score parts drives most of the weakness. Softening those parts when models are merged lifts scores. The finds give a map for making models more robust.

Catalog record

CREATORRunpeng Dai, Run Yang, Fan Zhou, Hongtu Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03714

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models Runpeng Dai, Run Yang, Fan Zhou, Hongtu Zhu Abstract Large Language Models (LLMs) and Vision-Language Models (VLMs) have achieved impressive performance across a wide range of tasks, yet they remain vulnerable to carefully crafted perturbations. In this study, we seek to pinpoint the sources of this fragility by identifying parameters and input dimensions (pixels or token embeddings) that are susceptible to such perturbations. To this end, we propose a stability measure called FI , F irst order local I nfluence, which is rooted in information geometry and quantifies the sensitivity of individual parameter and input dimensions. Our extensive analysis across LLMs and VLMs (from 1.5B to 13B parameters) reveals that: (I) A small subset of parameters or input dimensions with high FI values disproportionately contribute to model brittleness. (II) Mitigating the influence of these vulnerable parameters during model merging leads to improved performance. 1 Introduction 1 1 1 ∗ Both authors contributed equally to this work. Large Language Models (LLMs) and Vision Language Models (VLMs) such as GPT ( Brown et al., 2020 ) and Llama ( Touvron et al., 2023 ) , have revolutionized the field of Natural Language Processing (NLP), exhibiting remarkable proficiency across a variety of tasks ( Gong et al., 2024 ; Zheng et al., 2025b ; Luo et al., 2025 ) and modalities ( Bai et al., 2023 ; Liu et al., 2024b ; Zheng et al., 2025a ) . These modern LLMs are massive in size, trained on vast amounts of data, and meticulously aligned to prevent from generating harmful content ( Perez et al., 2022 ) , leaking private information ( Zhang et al., 2024 ) , or exhibiting sexual or religious bias ( Xie and Lukasiewicz, 2023 ) . Despite the enthusiasm for these integrative approaches, a critical issue remains: LLMs remain susceptible to both external and internal perturbations, affecting their reliability and performance. Externally , LLMs are vulnerable to input perturbations, such as Embedding-Corrupted Prompts ( Fort, 2023 ; Liu et al., 2024a ) . This susceptibility extends to visual inputs in VLMs, where adversarially optimized images can drastically alter model behavior ( Qi et al., 2024 ) . Beyond adversarial attacks, VLMs exhibit high sensitivity to perturbations in specific local regions of an image—a common issue, as user-uploaded images often suffer from blurring, masking, or low resolution. The vulnerability is highlighted in our case study of the Qwen-VL model. As depicted in Figure 1 , masking the ten most sensitive pixels, which are unrelated to the question, resulted in incorrect model outputs. Figure 1: A case study of the Qwen-VL model ( Bai et al., 2023 ) on SCI-QA. The image on the far right visualizes the per-pixel FI values. Masking just 10 pixels with the highest FI values leads to a failure in producing the correct answer. Internally , LLM stability is further challenged by parameter perturbations, often introduced through model merging and quantization. While these techniques improve deployment efficiency by reducing inference costs ( Frantar and Alistarh, 2023 ; Ashkboos et al., 2024 ) , they can also induce hallucinations and degrade performance ( Men et al., 2024 ; Yu et al., 2024 ; Li et al., 2024 ) . However, our findings reveal that parameter susceptibility varies significantly. As Figure 2 illustrates, randomly dropping 5% of parameters has a minimal impact on performance. In contrast, zeroing out just 1% of the parameters identified by our measure can drastically reduce accuracy, even below random guessing levels. Figure 2: A case study of Qwen2 on MMLU-geography. This figure illustrates the decline in response accuracy as a greater percentage of parameters are zeroed. “FI-High” targets parameters with the highest FI values, while “Random” indicates random parameter removal. To pinpoint the sources of this fragility, we propose a novel stability measure called FI , F irst order local I nfluence, to quantitatively assess the stability of LLMs against perturbations. Specifically, we construct a perturbation manifold that encompasses all perturbed models, along with its associated geometric properties. Our stability measure quantifies the degree of local influence of a perturbation on a given objective function within this manifold, thereby reflecting the stability of individual LLM components. FI’s versatility allows for effective stability assessment under both external and internal perturbations across various granularities—from individual parameters to input features like pixels and patches. FI effectively identifies vulnerabilities . Our extensive studies validate its effectiveness in pinpointing fragile pixels in VLM vision inputs, vulnerable embedding dimensions of tokens in LLMs (Section 4 ), and salient model parameters (Subsection 5.1 ). FI offers insights into improving model robustness . We further illustrate that understanding these vulnerabilities can lead to enhanced model resistance to perturbations. By focusing on model merging as an example, we show that safeguarding key parameters identified by high FI values can substantially reduce performance degradation during the merging process (Subsection 5.2 ). 2 Related Work Recent efforts to evaluate LLM stability typically adopt a coarse-grained approach, aiming to assess the overall robustness of models under various perturbations. One line of work investigates how stability is influenced by sampling parameters, such as temperature, which affect output variability during generation ( Atil et al., 2024 ; Ouyang et al., 2025 ) . Another direction studies model sensitivity to input or parameter perturbations. For instance, Liu et al. () analyze input-level robustness using optimal transport to quantify a model’s response to distributional shifts in prompts. On the other hand, Peng et al. (2024) focus on parameter-space perturbations, demonstrating that LLMs remain robust to weight changes up to a certain threshold, beyond which performance significantly degrades. They estimate a model’s robustness tolerance by injecting random perturbations into model weights and evaluating the performance drop. Despite these contributions, fine-grained analyses—such as those examining the effect of individual input tokens, pixels, or specific model parameters—remain underexplored. Wei et al. (2024) take a step in this direction by leveraging pruning-based techniques, including SNIP ( Lee et al., 2018 ) and Wanda ( Sun et al., 2023 ) , to identify critical neurons and low-rank structures that impact model safety and utility. However, there is still a lack of unified metrics or frameworks that assess stability with respect to both input- and parameter-level perturbations. Moreover, the downstream applications of such stability assessments remain largely unexamined. 3 Stability Measure of Large language models In this section, we propose a new metric called FI to quantify the stability of large language models against local perturbations. Considering the auto-regressive nature of LLMs, we first develop FI for single-step generation and discuss its theoretical and computational properties in detail. We then show how FI can be naturally extended to sequence generation tasks. Finally, we compare FI to existing stability measures, highlighting its unique advantages. 3.1 FI Metric Problem formulation. Consider an LLM parameterized by \theta , with input data x , which may consist of text or, for visual language models, a combination of text and images. Given x , the model generates a probability distribution over its vocabulary to predict the next token, which can be framed as a classification problem with K classes, where K represents the vocabulary size. However, vocabulary sizes are

Read the complete text