Don’t Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models

Shaotian Yan, Chen Shen, Wenxiao Wang, Liang Xie, Junjie Liu, Jieping Ye · 2025

Read Don’t Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models by Shaotian Yan, Chen Shen, Wenxiao Wang, Liang Xie, Junjie Liu, Jieping Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11154.txt.

Introduction

Shaotian Yan and co-authors find that single stray words in worked examples can derail a language model's reasoning, in a 2025 paper. Few shot chain of thought prompts guide a model through steps, yet some tokens in those demos grab the model's focus and pull in noise. The team traced attention inside the model step by step to find the culprits.

Tokens that look straight at the prediction states, without soaking up wider context, are the risky ones. Their method spots those tokens and softens their attention pull on the fly. Scores rose across benchmarks, with a near six point gain on one math word problem set.

Catalog record

CREATORShaotian Yan, Chen Shen, Wenxiao Wang, Liang Xie, Junjie Liu, Jieping Ye
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11154

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Don’t Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models Shaotian Yan, Chen Shen, Wenxiao Wang, Liang Xie, Junjie Liu, Jieping Ye Abstract Few-shot Chain-of-Thought (CoT) significantly enhances the reasoning capabilities of large language models (LLMs), functioning as a whole to guide these models in generating reasoning steps toward final answers. However, we observe that isolated segments, words, or tokens within CoT demonstrations can unexpectedly disrupt the generation process of LLMs. The model may overly concentrate on certain local information present in the demonstration, introducing irrelevant noise into the reasoning process and potentially leading to incorrect answers. In this paper, we investigate the underlying mechanism of CoT through dynamically tracing and manipulating the inner workings of LLMs at each output step, which demonstrates that tokens exhibiting specific attention characteristics are more likely to induce the model to take things out of context; these tokens directly attend to the hidden states tied with prediction, without substantial integration of non-local information. Building upon these insights, we propose a Few-shot Attention Intervention method (FAI) that dynamically analyzes the attention patterns of demonstrations to accurately identify these tokens and subsequently make targeted adjustments to the attention weights to effectively suppress their distracting effect on LLMs. Comprehensive experiments across multiple benchmarks demonstrate consistent improvements over baseline methods, with a remarkable 5.91% improvement on the AQuA dataset, further highlighting the effectiveness of FAI. 1 Introduction Large Language Models (LLMs) have achieved significant advancements in tackling complex reasoning tasks ( Zhou et al., 2023 ; Yao et al., 2023 ; Besta et al., 2023 ) , such as mathematics ( Imani et al., 2023 ; Cobbe et al., 2021 ; Yuan et al., 2023 ) and symbolic logic ( Patel et al., 2021 ; Srivastava et al., 2022 ; Ling et al., 2017 ) , by adopting the innovative Chain-of-Thought (CoT) prompting strategy ( Wei et al., 2022 ) which promotes the LLMs’ propensity to break down the thought process into multiple intermediary steps leading to the final answer. The most prevalent paradigm of CoT is known as few-shot CoT, which comprises a handful of demonstrations, each consisting of a query paired with a reasoning chain. However, in practice, the performance of LLMs is sensitive to the selection of CoT demonstrations ( Huang et al., 2023 ; Rubin et al., 2021 ; Luo et al., 2023 ; Liu et al., 2023 ; Su et al., 2022 ) . Employing diverse CoT exemplars can cause considerable variations in the overall precision of LLMs. We further demonstrate that even when overall accuracy rates are comparable, varying CoT demonstrations can lead to substantial differences in the distribution of specific questions that are answered correctly versus those answered incorrectly. This inconsistency raises concerns about the robustness of CoT and presents a crucial challenge for its real-world application. Yet the underlying cause of the observed performance variations remains largely unclear. Figure 1: An example of a Few-shot Chain-of-Thought demonstration distracting the reasoning of LLMs. Questions are collected from the GSM8K ( Cobbe et al., 2021 ) dataset, and the responses are generated by Llama-3-8B-Instruct ( AI@Meta, 2024 ) . Recent studies ( Madaan & Yazdanbakhsh, 2022 ; Tang et al., 2023 ; Wang et al., 2023a ; Jin et al., 2024 ; Ye et al., 2023 ; Fu et al., 2022 ; Prystawski et al., 2024 ) have made efforts on identifying the factors that influence the effectiveness of CoT demonstrations. ( Wang et al., 2023a ) shows that employing invalid reasoning steps within the CoT demonstrations can still yield performance that is nearly comparable to using valid reasoning. They further emphasize that the performance relies more critically on the logical consistency of the rationale and its relevance to the corresponding question. ( Jin et al., 2024 ) finds that increasing reasoning steps in the CoT demonstrations can improve the overall accuracy of LLMs across multiple datasets via contrastive experiments. However, the insights presented by these existing studies are primarily based on the overall accuracy across the entire dataset, failing to illuminate the instance-specific fluctuations inherent of CoT. While existing studies primarily focus on the positive effects of CoT, in this paper, motivated by the aforementioned fragilities of CoT, we find that few-shot CoT does not always perform in the way we expect. Specifically, in addition to the commonly known positive instructional effect of CoT demonstrations, which encourages LLMs to output intermediate reasoning steps leading to the final answer, certain localized information within the demonstrations can unexpectedly distract the reasoning process of LLMs in an undesirable manner. As shown in Figure 1 , the model inaccurately incorporates information from demonstrations (i.e., “each jar can hold 160 quarters” which is bolded and underlined in Figure 1 (c).) while generating intermediate reasoning steps in its output process, ultimately leading to wrong answers. In other cases, despite LLMs not directly copying information from demonstrations, their outputs are nonetheless implicitly distracted by specific pieces of information or tokens within the demonstrations. Ideally, LLMs should treat CoT demonstrations as a whole, learning and mimicking the implied style and higher-order semantics within them. They should apply this knowledge flexibly, rather than simply taking the content of the demonstrations out of context, memorizing it, or being distracted by it. Therefore, the distracting effect should be alleviated to enhance the robustness of CoT reasoning. Nevertheless, it is closely intertwined with the complete semantics of the demonstration. Efforts to simply remove some tokens may disrupt the overall meaning of the demonstration, thereby diminishing the effectiveness of the CoT demonstration. This dilemma prompts a deeper examination of the internal workings of LLMs. By visualizing the interactions, quantified through commonly used attention saliency scores, between tokens at each layer and time step, we observe that some tokens in the demonstrations have the following attention characteristics: these tokens do not significantly gather information from other tokens (meaning that the hidden state corresponding to these tokens retains a considerable amount of their own semantic information) but can directly channel towards the prediction position at certain time steps. The behavior of the cases depicted in Figure 1 (c) aligns closely with the information flow characteristics exhibited by these tokens, and indeed, we observe such features in the case presented in Figure 1 as well as in a quantitative experiment which involves about 180 other cases where the model is disrupted (see section 2 for detail). This mirrors human cognitive tendencies, where our attention is often disproportionately drawn towards salient local elements, leading to an inadvertent overemphasis on these aspects at the expense of more fundamental global context. Given their retention of substantial semantic integrity, these tokens are particularly susceptible to becoming prominent focal points under certain conditions, thereby influencing information processing dynamics. To address the aforementioned issue, we introduce a F ew-shot A ttention I ntervention (FAI) technique that dynamically analyzes the attention patterns of demonstrations to accurately identify tokens with rather isolated semantics. By making targeted adjustments to the attention weights, FAI can block the information flow from these tokens to the output token, effectively suppressing their distracting effect on LLMs. Comprehensive experimentation across various reasoning benchmarks demonstrates that with only lightweight and efficient interventions—comprising about 15% of the

Read the complete text