One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer

Jucheng Shen, Barbara Su, Anastasios Kyrillidis · 2026

Read One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer by Jucheng Shen, Barbara Su, Anastasios Kyrillidis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17811.txt.

Introduction

Jucheng Shen, Barbara Su, and Anastasios Kyrillidis ask whether one shared recurrent transformer can split into two roles without separate modules. In Asymmetric Input Recurrence, the same model updates two states, with input injected only during L-updates. Rollouts on Sudoku-Extreme and Maze show a split: the H state acts as a proposal, while the L state keeps uncertainty and structure. Freezing tests confirm the split tracks dynamics, and attention shows L-updates stay more local than H-updates. A signal telling update types apart induces the specialization.

Catalog record

CREATORJucheng Shen, Barbara Su, Anastasios Kyrillidis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52289
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17811

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer Jucheng Shen, Barbara Su, Anastasios Kyrillidis Abstract Can a shared-weight recurrent Transformer develop distinct internal roles without being partitioned into separate modules? We study this in Asymmetric Input Recurrence (AIR), a minimal two-state reasoning architecture in which the same Transformer model is reused for both updates (per literature, L and H) and the only built-in difference in the update rule is that the encoded input is injected during L-updates but not H-updates. Across Sudoku-Extreme and Maze, decoded rollouts reveal a consistent split: 𝐳 H \mathbf{z}_{H} behaves like a fully committed proposal state, whereas 𝐳 L \mathbf{z}_{L} retains local uncertainty and shifting intermediate structure. Freeze experiments show that this split is, in practice, related to the model’s state dynamics: in Sudoku, freezing 𝐳 H \mathbf{z}_{H} reduces 𝐳 L \mathbf{z}_{L} ’s content changes whereas freezing 𝐳 L \mathbf{z}_{L} increases 𝐳 H \mathbf{z}_{H} ’s, while in Maze, freezing either state increases content changes in the other state. Ablations show that to induce specialization, the shared model needs to be able to tell the two update types apart, either from input injection asymmetry or from a separate level token. Mechanistically, attention analysis shows that L-updates are consistently more local than H-updates in both Sudoku and Maze. Together, these results show that, in a two-state recurrent setting, a clear state-identity signal can induce stable, related functional roles inside a shared-parameter recurrent Transformer. Code is available at https://github.com/juchengshen/air . 1 Introduction Figure 1 : AIR testbed. The same shared Transformer f(\cdot;\mathbf{\theta}) is reused at two positions in the recurrent schedule: with the encoded input \tilde{\mathbf{x}} for L-updates (top) and without it for H-updates (bottom). Despite identical parameters, this input difference lets the shared model tell the two update types apart: \mathbf{z}_{L} acts like a shifting-commitment scratchpad while \mathbf{z}_{H} acts like a fully committed solution state, and each query token in L-updates attend to other tokens more locally than H-updates. Small recurrent reasoning models can perform surprisingly well on abstract reasoning tasks. HRM ( 23 ) uses two recurrent modules, while TRM ( 15 ) replaces them with a single shared-weight recursive network. Inspired by these models, we raise the following question: when the same model is reused for two latent states, does it learn distinct computational roles or merely process both states in the same way? The answer matters because it tests whether specialization needs separate parameter models, or whether one shared computation can specialize when it gets a stable signal telling it which state it is updating. To study this question, we introduce Asymmetric Input Recurrence (AIR), a minimal testbed that keeps TRM’s shared recurrent model but uses HRM’s one-step truncated gradient. In AIR, the only built-in difference in the update rule is whether the encoded input \tilde{\mathbf{x}} is injected ( Figure 1 ), so AIR is a controlled setting for testing whether this single difference in update content can induce a division of labor between a high-level and a low-level latent state. Long story short, our answer is yes for the cases this paper studies. Across Sudoku-Extreme ( 20 ) and Maze ( 19 ) , decoded rollouts show a stable split in which \mathbf{z}_{H} behaves like a committed proposal state and \mathbf{z}_{L} retains local uncertainty and shifting intermediate structure. Freeze experiments show that this split is related to the model’s state dynamics, and attention analysis reveals a matching mechanism: L-updates are more local and, on Sudoku, more violation-sensitive, whereas H-updates are more global. Our key contributions are as follows: 1. Existence of functional specialization. We demonstrate the existence of functional specialization in a shared-parameter recurrent Transformer. When decoding the intermediate output of each state, we observe that AIR develops a stable committed- \mathbf{z}_{H} / partially uncommitted- \mathbf{z}_{L} split, with a matching L-local versus H-global attention pattern ( Sections 4 and 7 ). 2. State-identity signal induces functional specialization. We empirically show that functional specialization emerges and improves final accuracy when the shared model can tell the two update types apart—either through input injection asymmetry or through a separate level token ( Section 5 ). 3. Two states are related. Last but not least, freeze experiments show that the latent states are related; in Sudoku, freezing \mathbf{z}_{H} reduces \mathbf{z}_{L} ’s content changes whereas freezing \mathbf{z}_{L} increases \mathbf{z}_{H} ’s, while in Maze, freezing either state increases content changes in the other state. ( Section 6 ). 2 Related Work Iterative and recurrent reasoning. AIR belongs to work that uses recurrence or weight-sharing to turn small networks into multi-step reasoners. Universal Transformers ( 6 ) and later iterated models showed that weight sharing can support flexible computation and adaptive depth ( 10 ; 21 ; 2 ) . Its closest predecessors are HRM ( 23 ) and TRM ( 15 ) : HRM assigns the two latent states to different parameter models, whereas TRM ties them to one shared model. AIR keeps the shared-model setting but asks whether distinct roles can still emerge inside that shared recurrence when input reaches at least one state, and when the model can tell which state it is updating. 3 show that input re-injection can prevent degeneration in looped language models; we ask whether asymmetric input injection can also help a shared model separate two recurrent roles. Modularity and emergent specialization. Our question is also about the distinction between structural modularity and functional specialization . Neural Module Networks ( 1 ) and Recurrent Independent Mechanisms ( 11 ) impose modular structure by design, whereas work on emergent modularity studies when differentiated behavior can arise inside initially monolithic models ( 5 ; 22 ) . AIR isolates this issue in a particularly strict setting: because the recurrent model is shared, any division of labor must emerge from training rather than architecture. Thus our novelty is not shared recurrence by itself, but showing, ablating, and probing the functional specialization that arises when a shared model can tell the two states apart. Mechanistic analysis and latent workspaces. Methodologically, our paper is closest to mechanistic and perturbation-based analyses of reasoning models ( 7 ; 17 ; 4 ; 14 ) , but we analyze recurrent latent states rather than individual heads or circuits. It also connects to work on internal workspaces for reasoning: unlike token-level scratchpads or chain-of-thought ( 16 ; 24 ) , AIR resembles latent-reasoning approaches such as COCONUT ( 13 ) , with \mathbf{z}_{L} behaving like a partial workspace and \mathbf{z}_{H} like a more committed proposal state. The difference is that AIR does not prescribe these roles; it asks whether they emerge in a shared-weight recurrent Transformer when the model can tell the two states apart. 3 Preliminaries We compare three closely related recurrent architectures: HRM, TRM, and our variant AIR. All maintain two latent states, \mathbf{z}_{H},\mathbf{z}_{L}\in\mathbb{R}^{B\times S\times D} , where B is batch size, S is sequence length, and D is hidden dimension. Let \tilde{\mathbf{x}} denote the encoded input. HRM ( 23 ) , TRM ( 15 ) , and AIR all use the following two-state recurrence in general; their specific differences are described below: \displaystyle\text{{L-update:}}\quad\mathbf{z}_{L} \displaystyle\leftarrow f_{L}\!\left(\mathbf{z}_{L}+\mathbf{z}_{H}+\alpha_{L}\tilde{\mathbf{x}};\,\mathbf{\theta}_{L}\right), (1) \displaystyle\text{{H-update:}}\quad\mathbf{z}_{H} \displaystyle\leftarrow f_{H}\!\left(\mathbf{z}_{H}+\mathbf{z}_{L}+\alpha_{H}\tilde{\mathbf{x}};\,\mathbf{\theta}_{H}\right). (2) For all three variants, we use the default input pattern (\alpha_{L},\alpha_{H})=(1,0) unless otherwise stated, so the

Read the complete text