A Mechanistic Analysis of Looped Reasoning Language Models

Hugh Blayney, Álvaro Arroyo, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Michael Bronstein, Xiaowen Dong · 2026

Read A Mechanistic Analysis of Looped Reasoning Language Models by Hugh Blayney, Álvaro Arroyo, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Michael Bronstein, Xiaowen Dong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11791.txt.

Introduction

Blayney and six colleagues examine looped reasoning language models, which reuse the same layers repeatedly instead of stacking fresh ones. They show each layer in the cycle settles on its own fixed point, so the recurrent block follows a steady cyclic path and attention behaviour stabilises across repeats. The stages of inference learned this way closely mirror those of ordinary feedforward models, repeated in depth with each loop, offering guidance for architectural design.

Catalog record

CREATORHugh Blayney, Álvaro Arroyo, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Michael Bronstein, Xiaowen Dong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79445
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11791

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Mechanistic Analysis of Looped Reasoning Language Models Hugh Blayney, Álvaro Arroyo, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Michael Bronstein, Xiaowen Dong Abstract Reasoning has become a central capability in large language models. Recent research has shown that reasoning performance can be improved by looping an LLM’s layers in the latent dimension, resulting in looped reasoning language models . Despite promising results, few works have investigated how their internal dynamics differ from those of standard feedforward models. In this paper, we conduct a mechanistic analysis of the latent states in looped language models, focusing in particular on how the stages of inference observed in feedforward models compare to those observed in looped ones. To this end, we analyze cyclic recurrence and show that for many of the studied models each layer in the cycle converges to a distinct fixed point; consequently, the recurrent block follows a consistent cyclic trajectory in the latent space. We provide evidence that as these fixed points are reached, attention-head behavior stabilizes, leading to constant behavior across recurrences. Empirically, we discover that recurrent blocks learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iteration. We study how recurrent block size, input injection, and normalization influence the emergence and stability of these cyclic fixed points. We believe these findings help translate mechanistic insights into practical guidance for architectural design. \printArXivAffiliationsAndNotice 1 Introduction The vast majority of current LLMs are based on the Transformer architecture ( 42 ) , which comprises a sequence of blocks traversed in a feedforward manner to predict the next token. As the capability of these models increased, attention turned to eliciting reasoning capabilities in LLMs by increasing test-time computation, commonly through chain-of-thought (CoT) prompting ( 45 ) or reinforcement-learning-based fine-tuning, first popularized in the DeepSeek-R1 architecture ( 21 ) . More recently, research has explored building reasoning capabilities directly into the model architecture via recurrent looping ( 18 ; 44 ; 24 ) , where additional test-time compute is spent by taking more recurrent steps, echoing early designs in this direction ( 19 ) . Despite growing empirical success, the mechanisms underlying these models remain poorly understood, as well as their benefits and limitations when compared to feedforward computation. Figure 1: Latent states after each block in a recurrent model frequently tend towards separate fixed points , meaning that the application of a recurrent block tends towards a consistent trajectory in latent space. In this paper, we compare how feedforward and looped LLMs organize computation across (effective) depth through the lens of stages of inference ( 28 ; 34 ) , a perspective suggesting that LLM inference can be decomposed into several distinct computational stages. Building on prior observations that repeated application of a shared recurrent block can approach a fixed point or steady state ( 48 ; 18 ) , we show that such behavior necessarily implies one of two possibilities: either the contribution of the component Transformer blocks vanishes asymptotically, or their sequential application traces out a constant cyclic trajectory in latent space. We further demonstrate empirically that the latter behavior arises in practice when certain architectural conditions are met, and that this appears to be emergent behavior from the Transformer architecture itself, appearing in both trained recurrent models and untrained, randomly initialized models. Our analysis yields two key insights: 1. Section 4 establishes that many looped language models tend toward cyclic fixed-point behavior, providing theoretical and empirical proof that this implies convergence to fixed attention patterns. Section 4.2 explores how specific architectural choices influence this behavior. 2. Section 5 demonstrates that these stable attention patterns mirror the “mixing” stages of inference learned in feedforward models. We then provide evidence in Section 5.1 that models naturally self-organize into these stages during training, and show in Section 5.2 that stable fixed-point models successfully maintain these inference stages, whereas unstable models deviate. 2 Preliminaries 2.1 Looped Transformers In this section we introduce Looped Transformers and define the notation that we will use throughout the paper. We represent the input sequence to our transformer of length T and dimension D as {\bm{X}}\in\mathbb{R}^{T\times D} . Following the notation of 49 we define \attn , the dot product self-attention mechanism, as a map f:\mathbb{R}^{T\times D}\to\mathbb{R}^{T\times D} : \displaystyle\attn({\bm{X}}) \displaystyle=\mathrm{softmax}{\left(\frac{{\bm{X}}{\bm{W}}_{Q}{\bm{W}}_{K}^{\top}{\bm{X}}^{\top}}{\sqrt{d}}\right)}{\bm{X}}{\bm{W}}_{V}, (1) \displaystyle=\mathrm{softmax}{\left(A({\bm{X}})\right)}{\bm{X}}{\bm{W}}_{V}, (2) where {\bm{W}}_{Q},{\bm{W}}_{K},{\bm{W}}_{V}\in\mathbb{R}^{D\times d} are projection matrices and A is defined for convenience. A transformer block typically comprises an attention mechanism and a position-wise MLP as follows: \displaystyle\hat{{\bm{X}}} \displaystyle=n_{2}\left({\bm{X}}+\attn(n_{1}({\bm{X}})\right), (3) \displaystyle{{\bm{X}}}^{\prime} \displaystyle=n_{4}\left(\hat{{\bm{X}}}+\text{MLP}(n_{3}(\hat{{\bm{X}}}))\right), (4) where n_{1},n_{2},n_{3},n_{4} are each optional norms – here we are borrowing from the notation of 18 . We denote the action of a Transformer block \block:\mathbb{R}^{T\times D}\to\mathbb{R}^{T\times D} as {\bm{X}}^{\prime}=\block({\bm{X}}) , and refer to the intermediate hidden-state matrices {\bm{X}} between blocks as the residual stream . Looped Transformers are Transformers that utilize “recurrence in depth” – that is, they reapply layers to repeatedly act on the latent states. Recent research has identified that an effective way ( 5 ) to achieve this is via cyclic recurrence : a fixed sequence of layers is repeated in a “cyclic” pattern. This is the approach that we focus on in this work, and we introduce it in more detail below. For convenience, we define a k -stacked block as a composition of Transformer blocks, \stack_{k}({\bm{X}})=\block_{k}(\block_{k-1}(\dots\block_{1}({\bm{X}})\dots)) . In the case of 18 ; 31 this stacked block may also take an additional input {\bm{Z}}\in\mathbb{R}^{T\times D} which is typically initialized from a Normal distribution, as well as the original input to the recurrent section: this is known as input injection ( 6 ; 2 ) , and the two inputs are projected into common feature space \mathbb{R}^{D} before the block is applied. In the case of input injection, a k -stacked block therefore becomes \stack_{k}({\bm{X}},{\bm{Z}})=\block_{k}(\block_{k-1}(\dots\block_{1}([{\bm{X}},{\bm{Z}}]{\bm{W}}_{I})\dots)), (5) where [\cdot,\cdot] denotes concatenation in the channel dimension and {\bm{W}}_{I}\in\mathbb{R}^{2D\times D} is a learned projection matrix. This allows us to define a (k\otimes l) -Recurrent block as a k -stacked block repeated l times: R_{l,k}({\bm{X}})=\overbrace{\stack_{k}(\stack_{k}(\dots\stack_{k}(}^{\times l}{\bm{X}})\dots)), (6) which with input-injection becomes R_{l,k}({\bm{X}},{\bm{Z}})=\overbrace{\stack_{k}({\bm{X}},\stack_{k}(\dots{\bm{X}},\stack_{k}({\bm{X}},{\bm{Z}}))\dots))}^{\times l}. (7) Note that the input {\bm{X}} is only “injected” at the start of each stack of blocks – once per recurrence. Additionally, a complete looped Transformer may have multiple feedforward layers before the Recurrent block, and multiple feedforward layers after the Recurrent block: following the convention of 18 , we refer to these as prelude and coda layers respectively, and these are simply separate stacked blocks with non-tied layer weights. Where prelude and coda are used, we will frequently refer to this as a sandwich block structure. We combine and adapt the nomenclature of 18 ; 36 and refer to a looped Transformer with p prelude layers, k recurrent layers and c coda layers with the tuple (p,k,c) . Where input injection is used, we add an I subscript (p,k,c)_{I} , and when referring to a recurrent layer looped a specific number of times l , we denote this as (p,k\otimes l,c) . In summary, a (p,k\otimes l,c) looped Transformer is defined as \displaystyle{\bm{X}}_{0} \displaystyle\leftarrow\stack_{p}({\bm{X}}) \displaystyle{\bm{X}}_{i} \displaystyle\leftarrow\stack_{k}^{\prime}({\bm{X}}_{i-1})\qquad i\in\{1,\dots,l\} \displaystyle{\bm{X}} \displaystyle\leftarrow\stack_{c}^{\prime\prime}({\bm{X}}_{l}), where \prime and \prime\prime indicates that these are different stacks between which weights are not shared. A (p,k\otimes l,c)_{I} looped Transformer (with input injection) is defined as \displaystyle{\bm{X}}

Read the complete text