Interpreting the Repeated Token Phenomenon in Large Language Models

Itay Yona, Ilia Shumailov, Jamie Hayes, Federico Barbero, Yossi Gandelsman · 2025

Read Interpreting the Repeated Token Phenomenon in Large Language Models by Itay Yona, Ilia Shumailov, Jamie Hayes, Federico Barbero, Yossi Gandelsman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08908.txt.

Introduction

Itay Yona and co-authors explain why large language models fail when asked to repeat one word many times, in a research paper from 2025. Instead of repeating, models drift into unrelated text. The failure is unexplained, and any user can trigger it, which makes it a real weakness in deployed systems.

The team links the failure to attention sinks, a pattern where the first token draws outsized attention. That pattern aids fluency, but long repeats break the circuit behind it. Other non-repeating sequences break the same circuit. A targeted patch fixes the repeat failure with no harm to general scores. The work shows how tracing circuits can repair model flaws.

Catalog record

CREATORItay Yona, Ilia Shumailov, Jamie Hayes, Federico Barbero, Yossi Gandelsman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08908

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interpreting the Repeated Token Phenomenon in Large Language Models Itay Yona, Ilia Shumailov, Jamie Hayes, Federico Barbero, Yossi Gandelsman Abstract Large Language Models (LLMs), despite their impressive capabilities, often fail to accurately repeat a single word when prompted to, and instead output unrelated text. This unexplained failure mode represents a vulnerability , allowing even end-users to diverge models away from their intended behavior. We aim to explain the causes for this phenomenon and link it to the concept of ‘‘attention sinks’’, an emergent LLM behavior crucial for fluency, in which the initial token receives disproportionately high attention scores. Our investigation identifies the neural circuit responsible for attention sinks and shows how long repetitions disrupt this circuit. We extend this finding to other non-repeating sequences that exhibit similar circuit disruptions. To address this, we propose a targeted patch that effectively resolves the issue without negatively impacting the model’s overall performance. This study provides a mechanistic explanation for an LLM vulnerability, demonstrating how interpretability can diagnose and address issues, and offering insights that pave the way for more secure and reliable models 1 1 1 Code: https://github.com/yossigandelsman/attn_sinkhole . \paperurl 1 Introduction Large Language Models (LLMs) have achieved remarkable success in various natural language tasks. However, even state-of-the-art LLMs can exhibit surprising failures on seemingly simple tasks. A prominent example is the “repeated token divergence” phenomenon ( Nasr et al., 2023 ; Barbero et al., 2025 ) , where LLMs struggle to accurately repeat a single input token when prompted to do so. This issue has been observed across a range of models, including: ChatGPT, LLaMa1, LLaMa2 ( OpenAI, 2022 ; Touvron et al., 2023a ; Touvron et al., 2023b ) . The underlying cause of this unexpected behavior has remained an open question up until now. In this paper, we explain the reason behind the repeated token divergence in LLMs. We link this behavior to the mechanism of “attention-sinks” – a phenomenon where the initial token in a sequence receives disproportionately high attention, previously shown to be crucial for LLM fluency ( Xiao et al., 2024 ) . To demonstrate the link between attention sinks and repeated token divergence, we take a Mechanistic Interpretability approach, analyzing the underlying mechanism of attention sinks in LLMs. We identify two key stages consistently observed across many models: First, the initial attention layer identifies and “marks” the first token in the sequence. Second, a single neuron in a later layer, triggered by this mark, adds high-magnitude values to the hidden state of the first token, creating the attention-sink. This high-magnitude hidden state then attracts attention in subsequent layers, as described by Xiao et al. (2024) . We present empirical evidence for this two-stage mechanism across multiple LLMs. Next, we examine how this mechanism interacts with repeating tokens. We find that the first attention layer, aiming to identify the first token, fails to distinguish it from a sequence of identical tokens. Consequently, it marks both the first token and the repeated tokens, activating neurons that amplify their hidden states. This leads to abnormally high attention weights on the repeated tokens, causing the model’s behavior to diverge and produce unexpected outputs. By analyzing the first attention layer implementation of identifying the first token we were able to generate a new exploit we call “ cluster attack”, that induces attention sinks without using repeated tokens. This exploit would be able to bypass surface level defenses that addresses repetitions but not underlying vulnerability. Finally, our analysis of the underlying mechanism suggests a simple yet effective correction to mitigate the repeating tokens divergence and its extensions. Our contributions are: 1. Mechanistic Explanation: We provide the first mechanistic explanation for the repeated token divergence in LLMs, linking it to the attention-sink phenomenon and identifying the responsible neural computation. We pinpoint a two-stage mechanism for attention sinks: (1) first-token marking by the initial attention layer, and (2) hidden-state amplification by specific MLP neurons ( Section 4 ). 2. Divergence Extension (Cluster Attack): We introduce a novel “cluster attack” demonstrating that the divergence extends beyond exact repetition to include similar tokens, revealing a broader vulnerability ( Section 5.2 ). 3. Targeted Mitigation: We develop a targeted correction method that effectively mitigates the divergence and the cluster attack without impacting performance on other tasks ( Section 5.3 ). 2 Related Work First, we discuss previous approaches that mechanistically interpreting various model behaviors. Then, we present related work about the two model behaviors we aim to mechanistically interpret - repeated token divergence and attention sinks in LLMs. 2.1 Mechanistic Interpretability of LLMs Mechanistic Interpretability aims to reverse engineer neural networks and uncover the underlying algorithms and computational mechanisms that drive their behavior ( Wang et al., 2022 ; Cunningham et al., 2023 ; Gandelsman et al., 2025 ; Jiang et al., 2025 ) . This field seeks to move beyond simply observing correlations to identifying causal relationships within the network, providing a deeper understanding of how these complex systems function ( Olsson et al., 2022 ; Olah et al., 2020 ) . Our work falls under this umbrella by dissecting the specific neural circuits responsible for both attention-sinks and the repeated tokens divergence in LLMs. 2.2 Repeated Tokens Divergence The phenomenon of LLMs failing to accurately repeat tokens was first described by Nasr et al. (2023) . While they observed the convergence of repeating token representations towards the BoS token representation, they offered only intuitive explanations for the matter calling for future work to investigate this further. Our work builds upon this observation by identifying the specific neural mechanisms and attention dynamics responsible for this divergence. We link this behavior to the attention-sink mechanism and demonstrate how the interaction between these two phenomena leads to the observed failures in token repetition. Barbero et al. (2025) explored the theoretical challenges of repeated tokens in LLMs (using an infinite context window) and provided empirical observations, but did not explicitly connect the two. Differently from previous work we present a different level of analysis ( Marr and Poggio, 1976 ) describing the algorithm and implementation of the mechanism that give rise to this issue. 2.3 Attention Sinks in LLMs LLM Attention Sinks were studied in multiple contexts. Xiao et al. (2024) showed that attention sinks relate to model fluency. Gu et al. (2024) explored why and how attention-sinks emerge. Sun et al. (2024) show the relationship of attention sinks to massive activations. Cancedda (2024) analyze what signals propagates through attention-sinks. Yu et al. (2024) propose a method to address the existence attention sinks. Bondarenko et al. (2024) ; Miller (2023) explored methods to mitigate attention sinks for model quantization and efficiency. However, our work takes a different approach, focusing on the functional role of attention sinks and their unintended consequences, specifically their connection to the repeated token divergence phenomenon. We investigate how the very mechanism that makes attention sinks can also be exploited. 3 Preliminaries This section provides a brief overview of the key concepts related to Large Language Models (LLMs) and attention sinks, setting the stage for our analysis of the repeated token divergence. Figure 1 : Attention scores for layers 2, 3, 17, and 31 of LLaMA2-7B-HF. As can be seen in the figure, the attention scores

Read the complete text