How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark

Minglai Yang, Ethan Huang, Liang Zhang, Mihai Surdeanu, William Wang, Liangming Pan · 2025

Read How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark by Minglai Yang, Ethan Huang, Liang Zhang, Mihai Surdeanu, William Wang, Liangming Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18761.txt.

Introduction

Minglai Yang and five co-authors ask how irrelevant context distracts language model reasoning in two thousand twenty-five. Their benchmark, GSM-DC, grows grade school math with planted distractors. Symbolic reasoning graphs let each distractor land in a set spot. Tests can thus be strict and repeatable across models.

Models proved sensitive to the planted noise in their tests. Both path choice and arithmetic accuracy took hits. Training with strong distractors lifted scores in and out of distribution. A stepwise tree search with a process reward helped more. Out of distribution strength gained the most from that search.

Catalog record

CREATORMinglai Yang, Ethan Huang, Liang Zhang, Mihai Surdeanu, William Wang, Liangming Pan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18761

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark Minglai Yang, Ethan Huang, Liang Zhang, Mihai Surdeanu, William Wang, Liangming Pan Abstract We introduce Grade School Math with Distracting Context (GSM-DC 1 1 1 The code of our dataset and experiment can be viewed at https://github.com/yminglai/GSM-DC . ), a synthetic benchmark to evaluate Large Language Models’ (LLMs) reasoning robustness against systematically controlled irrelevant context (IC). GSM-DC constructs symbolic reasoning graphs with precise distractor injections, enabling rigorous, reproducible evaluation. Our experiments demonstrate that LLMs are significantly sensitive to IC, affecting both reasoning path selection and arithmetic accuracy. Additionally, training models with strong distractors improves performance in both in-distribution and out-of-distribution scenarios. We further propose a stepwise tree search guided by a process reward model, which notably enhances robustness in out-of-distribution conditions. 1 Introduction Recent advances in Large Language Models (LLMs) have demonstrated reasoning capabilities across diverse tasks, notably in solving mathematical problems Cobbe et al. (2021) ; Lewkowycz et al. (2022) ; Zhou et al. (2022) ; Yao et al. (2023a) . Despite these advancements, LLMs are found to be less robust in reasoning Berglund et al. (2024) ; Huang et al. (2024) ; Xu et al. (2024) . For example, the Flanker Task Eriksen and Eriksen (1974) in cognitive psychology shows that humans’ responses become slower and less accurate with increased distractors. Shi et al. (2023a) first revealed that LLMs similarly suffer performance degradation when irrelevant context is introduced, observing notable reductions in accuracy even with just a single distractor sentence added to math problems from the GSM8K dataset Cobbe et al. (2021) . Figure 1: Overview of the GSM-DC framework: both generation and evaluation pipeline. The dataset construction process (orange) involves three key steps: (1) Dependency Graph Construction builds a symbolic DAG with a defined solution path via topological sort, (2) Irrelevant Context Injection adds controllable distractor nodes to increase reasoning complexity, and (3) Natural Language Realization converts the symbolic graph into a human-readable word problem and finds the solution following the solution path. The resulting instance is evaluated using a Stepwise Solution Evaluator that computes Step Accuracy, Path Accuracy, and Extraction Answer Accuracy. Prior work has not systematically explored the mechanisms underlying this sensitivity. Shi et al. (2023a) employed only a single distractor, limited experiments to short reasoning chains, and omitted supervised fine-tuning and out-of-distribution (OOD) evaluations. Several important questions thus remain: How does varying the amount of IC affect robustness? Can robust reasoning be enhanced through supervised fine-tuning? How does the intensity of IC during training impact model performance in both in-distribution and OOD scenarios? Additionally, how can the above questions be qualitatively evaluated? To address these gaps, we introduce GSM-DC , a synthetic benchmark designed to enable precise control over both reasoning complexity and distractor structure. Problems in GSM-DC are represented as symbolic dependency graphs, where nodes correspond to intermediate quantities and edges represent symbolic operations. This structure facilitates: 1) the explicit injection of irrelevant context via off-path nodes and edges without affecting correct solutions; 2) adjustment of reasoning complexity by varying graph depth and structure; and 3) automatic evaluation of model outputs by aligning predictions with the correct reasoning path. Our dataset construction pipeline ( Figure 1 ) involves generating symbolic dependency graphs, injecting distractors after determining the solution path, and transforming these into human-readable math word problems and solutions. We partition our dataset based on different problem complexities and distractor intensities, conduct various controlled experiments, and use automatic stepwise metrics measuring arithmetic correctness and distraction robustness. Our controlled experiments yield three main findings. First, model accuracy steadily decreases as distractor intensity rises. Second, continued pretraining substantially enhances reasoning robustness. Third, incorporating strong IC during training significantly boosts model resilience, showing superior performance across various distractor intensities in out-of-domain testing. To improve the model’s robustness against IC, we propose a stepwise beam search algorithm guided by a Process Reward Model (PRM), which scores partial reasoning paths based on their alignment with valid solution trajectories. This approach further improves robustness by up to 6.29% in out-of-domain conditions, highlighting reinforcement learning’s potential to strengthen robustness against irrelevant context in model reasoning. 2 Related Work Reasoning with Irrelevant Context LLMs often struggle to reason accurately in the presence of irrelevant context (IC). Prior work has explored this vulnerability by introducing distractors into math problems. For example, GSM-IC ( Shi et al., 2023a ) appends irrelevant sentences to arithmetic questions but lacks control over distractor structure or complexity. GSMIR ( Jiang et al., 2024a ) and MPN ( Song and Tavanapong, 2024 ) use handcrafted prompting strategies to mitigate the effects of textual noise. Anantheswaran et al. (2024) generate adversarial math problems by adding irrelevant variables, showing significant performance drops and partial robustness gains through fine-tuning. However, their hand-crafted distractors risk introducing bias and lack structural control. Other studies ( Wu et al., 2024 ; Wang et al., 2025b ; Xiao et al., 2025 ; Dong et al., 2025a ) show that semantically similar but irrelevant documents can impair LLM performance, but RAG can improve robustness ( Jiang et al., 2024b ; Dong et al., 2024 ; Liu et al., 2024 ; Wei et al., 2025 ) . While these works expose LLMs’ sensitivity to IC, they provide limited control over distractor properties. In contrast, GSM-DC injects distractors into symbolic reasoning graphs, enabling stepwise evaluation. We further show that a reward-guided beam search improves robustness beyond standard fine-tuning. Understanding LLM Reasoning LLM reasoning has received growing attention, leading to diverse efforts to improve performance on complex tasks. Recently, synthetic benchmarks such as GSM- \infty ( Zhou et al., 2025 ) and iGSM ( Ye et al., 2025 ) explored LLM reasoning under long-context and complex distractors. Unlike GSM- \infty and iGSM, our GSM-DC explicitly controls irrelevant distractors within symbolic DAGs to systematically quantify the effects of irrelevant context. Hao et al. (2024) introduced AutoRace and the LLM Reasoners library to standardize reasoning evaluation. CoT prompting and in-context learning ( Xiong et al., 2025 ; Yang et al., 2025 ) have been shown to enhance logical reasoning ( Bertolazzi et al., 2024 ) , while other work highlights limitations in handling strict deductions ( Li et al., 2024 ) . Recent methods such as ReAct ( Yao et al., 2023c ) , Tree-of-Thoughts ( Yao et al., 2023b ) , and self-consistency decoding ( Wang et al., 2023 ) guide intermediate steps to improve solution quality. Beyond final-answer supervision, Process Reward Models (PRMs) ( Uesato et al., 2022 ; Lightman et al., 2024 ; Zheng et al., 2024 ; Kumar et al., 2024 ; Hosseini et al., 2024 ) evaluate partial reasoning paths to promote more robust, interpretable, and aligned multi-step reasoning. Furthermore, Self-taught reasoners ( Zelikman et al., 2022 ; Huang et al., 2024 ) reallocates sampling toward boundary problems via reward-guided difficulty estimation. Finally, Shi et al. (2023b) showed that few-shot abduction boosts generalization with minimal supervision. 3 The GSM-DC Dataset To systematically investigate how LLMs reason under irrelevant context (IC), we require a framework that satisfies three desiderata: 1) fine-grained manipulation of IC, 2) precise control over reasoning difficulty, and 3) automatic evaluation of reasoning robustness. Existing datasets (§ 2 ) like

Read the complete text