Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics
Pankaj Kumar [email protected], Subhankar Mishra [email protected] · 2025
Read Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics by Pankaj Kumar [email protected], Subhankar Mishra [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18658.txt.
Introduction
Pankaj Kumar and Subhankar Mishra survey how to make large language models robust in a 2025 paper. A robust model gives steady, sound answers across many inputs, even when inputs are odd or hostile. Failures in live use can mislead users, so the stakes are real for work and public tools.
The survey first sets out what robustness means and how models fail. It then sorts the sources of weakness into flaws in the model, flaws in the data, and attacks from outside. Next it reviews the main fixes on offer, and the tests and scores used to judge them. The authors close with gaps in current testing and paths for new work.
Catalog record
| CREATOR | Pankaj Kumar [email protected], Subhankar Mishra [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 120 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18658
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics Pankaj Kumar [email protected], Subhankar Mishra [email protected] Abstract Large Language Models (LLMs) have emerged as a promising cornerstone for the development of natural language processing (NLP) and artificial intelligence (AI). However, ensuring the robustness of LLMs remains a critical challenge. To address these challenges and advance the field, this survey provides a comprehensive overview of current studies in this area. First, we systematically examine the nature of robustness in LLMs, including its conceptual foundations, the importance of consistent performance across diverse inputs, and the implications of failure modes in real-world applications. Next, we analyze the sources of non-robustness, categorizing intrinsic model limitations, data-driven vulnerabilities, and external adversarial factors that compromise reliability. Following this, we review state-of-the-art mitigation strategies, and then we discuss widely adopted benchmarks, emerging metrics, and persistent gaps in assessing real-world reliability. Finally, we synthesize findings from existing surveys and interdisciplinary studies to highlight trends, unresolved issues, and pathways for future research. 1 Introduction Large Language Models (LLMs), mainly characterized by their vast number of parameters, have emerged as a promising cornerstone for the development of natural language processing(NLP) and artificial intelligence (AI) Zhang et al. 2023 . These models have demonstrated remarkable capabilities across a wide range of NLP applications. Foundation models like GPT-4 OpenAI et al. 2024 , GPT-3 Brown et al. 2020 , Qwen 2.5-VL Bai et al. 2025a , Deepseek-V3 DeepSeek-AI et al. 2025 , Meta (LLaMA-3, LLaMA-2 Touvron et al. 2023b , LLaMA-1 Touvron et al. 2023a ), Gemini DeepMind 2023 are pre-trained on massive datasets and serve as the backbone for various AI tasks, from text generation to multimodal understanding—the ability to process and interpret multiple types of data, such as text and images. Their extensive factual knowledge has significantly improved their effectiveness in information retrieval, enabling them to provide more accurate and contextually relevant responses AlKhamissi et al. 2022 ; Petroni et al. 2019 . As a result, these models achieve high accuracy on benchmark evaluations, demonstrating their strong predictive capabilities across diverse tasks OpenAI et al. 2024 ; Bai et al. 2025a ; DeepSeek-AI et al. 2025 ; Grattafiori et al. 2024 . However, accuracy alone is insufficient to ensure reliability in real-world applications. In recent years, despite improvements in the accuracy of deep neural network models, researchers have found that they are easy to fool by applying a specific imperceptible perturbation Szegedy et al. 2014 . A deep learning model’s accuracy describes how accurately the model predicts sample points over a distribution. If a model cannot predict accurately, then no matter what properties this model holds, it would be meaningless. Although accuracy is the most fundamental metric in the evaluation and is essential to validate the performance of an LLM before it is released, it is equally important to test its robustness when deploying for the development of reliable AI systems. Robustness, a crucial aspect of deep learning models, describes their ability to maintain stable predictions when faced with specific perturbations or variations in the input data Liu et al. 2024b . Despite recent advancements, ensuring the robustness of LLMs remains a critical challenge. LLMs may encounter unpredictable variations in language, shifts in data (changes in the data patterns that a model encounters, which may affect its performance), and adversarial inputs that can affect their performance. In practical applications, these models often face noisy, unstructured text, breaking the assumption that input data will always be clean and well-formed. Additionally, a lack of robustness may result in unintended biases, incorrect predictions, or overreliance on spurious correlations (misleading correlations in data that a model may rely on incorrectly), raising concerns about their reliability in high-stakes environments such as healthcare Wan et al. 2024b ; Wan et al. 2024a ; He et al. 2025 , law Lee 2023 ; Cui et al. 2024 ; Yue et al. 2023 ; Bhambhoria et al. 2024 ; Bai et al. 2021b , complex reasoning Yang et al. 2023 , code generation Sun et al. 2024b ; Lin et al. 2024a ; Mishra et al. 2024 ; Hassid et al. 2024 , and finance Lee et al. 2025 ; Zhao et al. 2024a ; Li et al. 2024c ; Wu et al. 2023 . Because of this, a thorough and exacting assessment of LLM robustness that goes beyond conventional accuracy criteria is required. {forest} Figure 1: The outline of the survey on robustness of large language models. However, advancing research on the robustness of LLMs also faces significant challenges. First, the absence of standardized definitions and categorizations of robustness complicates cross-study comparisons, as research efforts often differ in how they frame their scope and components. Second, pinpointing the root causes of non-robustness, such as data biases, unstable training dynamics, architectural limitations, or inference inefficiencies, requires disentangling complex interactions across these factors. Third, while numerous mitigation strategies exist, their effectiveness often depends on addressing specific vulnerabilities, making it difficult to generalize solutions. Fourth, evaluating robustness remains fragmented, with no unified metrics or benchmarks to holistically assess performance under diverse real-world conditions like noisy inputs or adversarial attacks. Finally, existing literature on LLM robustness ( Zhao et al. 2025 ; Gu et al. 2025 ; He et al. 2025 ; Liu et al. 2024b ; Mehrabi et al. 2022 ; Wang et al. 2025 ; Lu et al. 2024 ; Weng 2023 ; Yang et al. 2024a ) is scattered across domains, limiting cross-pollination of insights and hindering the development of cohesive solutions. To address these challenges and advance the field, this survey provides a comprehensive overview of current studies in this area. Figure 1 shows the outline of this survey. We begin by clarifying definitions and categorizing the dimensions of robustness to establish a shared framework for analysis (§ 2 ). Next, we systematically identify and classify the primary sources of non-robustness, including data quality issues, training instabilities, architectural constraints, and inference-time vulnerabilities (§ 3 ). We then review state-of-the-art mitigation strategies, mapping each approach to the specific weaknesses it targets (§ 4 ). Following this, we analyze evaluation methodologies, discussing widely adopted benchmarks, emerging metrics, and persistent gaps in assessing real-world reliability (§ 5 ). Finally, we synthesize findings from existing surveys and interdisciplinary studies to highlight trends, unresolved issues, and pathways for future research (§ 7 ). By unifying these perspectives, we aim to foster collaboration and accelerate progress toward building more reliable, resilient LLMs. 2 Robustness in LLMs In the context of LLMs from various literatures, robustness refers to their ability to perform reliably and safely despite challenges like input variations, unexpected data, or adversarial attacks. It is frequently discussed as a critical component within the broader framework of AI Trustworthiness. Trustworthy AI systems are expected to be reliable, safe, fair, transparent, accountable, and robust. Robustness underpins several of these qualities; for instance, a model prone to adversarial manipulation (lack of robustness) cannot be considered safe or reliable ( Wang et al. 2023 ; Aljohani et al. 2025 ) . Similarly, a model that hallucinates or provides inconsistent answers under slightly different prompts lacks reliability, another facet closely tied to robustness ( Zhou et al. 2024b ) . Research into the vulnerabilities