Distractor Injection Attacks on Large Reasoning Models: Characterization and Defense

Zhehao Zhang, Weijie Xu, Shixian Cui, Chandan K. Reddy · 2025

Read Distractor Injection Attacks on Large Reasoning Models: Characterization and Defense by Zhehao Zhang, Weijie Xu, Shixian Cui, Chandan K. Reddy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16259.txt.

Introduction

Zhehao Zhang and three co-authors expose a vulnerability in large reasoning models: hide a hard, irrelevant problem inside the prompt and the model may chase it instead of the real task. They name this failure reasoning distraction and measure how badly it hurts state-of-the-art systems across models and benchmarks.

Injected distractors cut task accuracy by as much as sixty percent. Some alignment techniques make the weakness worse, and models sometimes follow hidden adversarial instructions inside their reasoning while concealing that compliance in the final answer, a covert obedience the authors document. Their defense trains on synthetic adversarial data with supervised fine-tuning plus reinforcement learning, recovering over fifty points of robustness. The paper frames distraction as a distinct, urgent threat to reasoning reliability.

Catalog record

CREATORZhehao Zhang, Weijie Xu, Shixian Cui, Chandan K. Reddy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16259

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distractor Injection Attacks on Large Reasoning Models: Characterization and Defense Zhehao Zhang, Weijie Xu, Shixian Cui, Chandan K. Reddy Abstract Recent advances in large reasoning models (LRMs) have enabled remarkable performance on complex tasks such as mathematics and coding by generating long Chain-of-Thought (CoT) traces. In this paper, we identify and systematically analyze a critical vulnerability we term reasoning distraction , where LRMs are diverted from their primary objective by irrelevant yet complex tasks maliciously embedded in the prompt. Through a comprehensive study across diverse models and benchmarks, we show that even state-of-the-art LRMs are highly susceptible, with injected distractors reducing task accuracy by up to 60%. We further reveal that certain alignment techniques can amplify this weakness and that models may exhibit covert compliance , following hidden adversarial instructions in reasoning while concealing them in the final output. To mitigate these risks, we propose a training-based defense that combines Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on synthetic adversarial data, improving robustness by over 50 points on challenging distractor attacks. Our findings establish reasoning distraction as a distinct and urgent threat to LRM reliability and provide a practical step toward safer and more trustworthy reasoning systems. Huggingface Data Collection: SFT and Preference Data Figure 1: An overview of the reasoning distraction setting, where we examine whether LRMs can be diverted from their primary objective by irrelevant or maliciously embedded tasks in the prompt. 1 Introduction The capabilities of LLMs in natural language understanding and complex reasoning have recently advanced with the emergence of models specifically designed for reasoning. This new class of models, often termed large reasoning models (LRMs) ( Xu et al. 2025a ) , such as OpenAI’s o1 ( Jaech et al. 2024 ) and DeepSeek-R1 ( Guo et al. 2025a ) , demonstrate substantial performance improvements on System-2 reasoning tasks ( Li et al. 2025b ) . A key mechanism behind this advancement is their ability to generate extensive CoT reasoning traces. These models articulate their thought processes, often encapsulated within nested tags like <think> and </think> , to explicitly outline behaviors such as planning and self-verification. This structured reasoning process enables LRMs to excel at solving challenging problems, particularly in domains such as mathematics and coding. Figure 2: An illustration of a reasoning distraction attack in which a malicious distractor task manipulates the reasoning of an LRM-as-a-judge, compelling it to select an attacker-specified response. Although a growing body of work has focused on improving reasoning efficiency by reducing the computational overhead of verbose output, a phenomenon known as overthinking ( Sui et al. 2025 ; Chen et al. 2024 ; Han et al. 2024 ) , far less attention has been paid to vulnerabilities arising from distractions in the reasoning process. Intuitively, the internal procedures, such as self-verification, run inside the CoT reasoning tokens and can draw on irrelevant information, which may harm a model’s robustness. For instance, in an LRM-as-a-judge setting, the model is tasked with determining which of two responses to a given prompt is superior. As shown in Figure 2 , a malicious adversary can insert an unrelated, complex math problem into one of the responses, instructing the LRM to award a higher score if it solves the problem. Our findings show that even SOTA LRMs such as DeepSeek-R1 can be easily diverted by such instructions, undermining the integrity of the evaluation process. We refer to this vulnerability as reasoning distraction , which differs from inefficiency (overthinking) and standard prompt injection. This paper provides the first systematic study of reasoning distraction in LRMs, as illustrated in Figure 1 . First, we conduct a comprehensive evaluation across a suite of models, testing their robustness against a diverse set of distractor tasks. Our findings reveal a widespread and critical vulnerability: even a wide range of SOTA LRMs are easily distracted by reasoning distractors. Our in-depth analysis further shows that post-training techniques like Reinforcement Learning with Verifiable Rewards (RLVR) can amplify distraction susceptibility. We also observe “covert compliance”: the model follows distractor instructions in its CoT while hiding that manipulation in its final output. To enhance robustness against reasoning distraction, we introduce a training-based mitigation strategy. By constructing a carefully curated dataset and applying the SFT and DPO training regimen, we demonstrate that models can learn to identify and ignore malicious distractors. Our results show this approach significantly enhances resilience, offering a promising recipe for building safer and more reliable LRMs. The main contributions of our work are: • We introduce and formalize reasoning distraction attacks , a new class of adversarial prompt manipulations that hijack LRMs’ chain-of-thought reasoning, exposing a critical and previously overlooked robustness gap. • We conduct the first systematic and large-scale empirical study , evaluating diverse LRMs across multiple distractor categories (math, coding, logic, etc.) and downstream tasks including mathematics, coding, knowledge understanding, instruction following, tool-use (agents), and LRM-as-a-judge, with ablations on factors such as distractor position. • We show that state-of-the-art LRMs are alarmingly vulnerable , with injected distractors reducing task accuracy by up to 60% and revealing novel failure modes such as Covert Compliance , where models secretly follow malicious instructions while hiding them in the final output. • We propose an effective defense framework based on synthetic adversarial data fine-tuning, combining Supervised Fine-Tuning (SFT) with preference-based Reinforcement Learning (RL), achieving over 50-point robustness gains on challenging distractors and establishing a practical path toward distraction-resilient LRMs. 2 Related Work Overthinking in LRMs. Recent work studies overthinking in LRMs: models perform unnecessary, repeated reasoning during inference ( Pu et al. 2025 ; Liu et al. 2025 ; Chen et al. 2024 ; Fu et al. 2024 ) . It is often detected by repeated intermediate answers ( Chen et al. 2024 ) and increases cost and latency (e.g., re-checking 1+2=3 ). Mitigations include mechanistic methods like Manifold Steering that steer activations to cut redundancy ( Huang et al. 2025 ) , and representational analyses of inefficiency ( Baek & Tegmark 2025 ; Chen et al. 2025a ) . Kumar et al. 2025 also views overthinking as an attack vector : simple decoy tasks inserted into RAG can slowdown reasoning. Prior work thus frames overthinking mainly as inefficiency . In contrast, we study reasoning distraction , where adversarial reasoning not only wastes compute but also degrades accuracy on the main task. Prompt Injection and Jailbreak Attacks. Prompt injection and jailbreaks are well-known LLM risks. Small adversarial suffixes or hidden context can make models ignore instructions, produce harmful text. For example, Benjamin et al. 2024 finds a broad vulnerability and Guo et al. 2025b shows that even trivial injections into benign text can break performance on simple multiple choice tasks. Prior work mainly studies direct command-style attacks. In contrast, we examine injections that embed complex reasoning tasks , which hijack the model’s chain-of-thought and bias its output. LLM-as-a-Judge Robustness. As LRMs are increasingly used as evaluators in benchmarking, alignment, and reinforcement learning from AI feedback (RLAIF), their reliability has been questioned. Maloyan et al. 2025 identifies vulnerabilities such as comparative undermining and justification manipulation attacks, while the RobustJudge benchmark ( Li et al. 2025a ) shows LLM-as-a-Judge’s robustness depends heavily on prompt templates and model choice, which highlights evaluator fragility under shallow

Read the complete text