Adaptive Loops and Memory in Transformers: Think Harder or Know More?
Markus Frey, Behzad Shomali, Ali Hamza Bashir, David Berghaus, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] · 2026
Read Adaptive Loops and Memory in Transformers: Think Harder or Know More? by Markus Frey, Behzad Shomali, Ali Hamza Bashir, David Berghaus, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08391.txt.
Introduction
Frey and colleagues ask if transformers think harder, know more, or do both. Chain-of-thought prompting lets models reason by stating steps in words. Looped transformers refine representations, saving parameters but losing storage, since layers get their own weights. Models use adaptive per-layer looping, where each block learns when to stop, plus memory banks. Looping helps math reasoning most, memory banks restore commonsense performance, and they beat a FLOPs-matched baseline with triple layers on benchmarks. Early layers loop little and use memory, while later do more.
Catalog record
| CREATOR | Markus Frey, Behzad Shomali, Ali Hamza Bashir, David Berghaus, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29964 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08391
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adaptive Loops and Memory in Transformers: Think Harder or Know More? Markus Frey, Behzad Shomali, Ali Hamza Bashir, David Berghaus, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] Abstract Chain-of-thought (CoT) prompting enables reasoning in language models but requires explicit verbalization of intermediate steps. Looped transformers offer an alternative by iteratively refining representations within hidden states. This parameter efficiency comes at a cost, as looped models lack the storage capacity of deeper models which use unique weights per layer. In this work, we investigate transformer models that feature both adaptive per-layer looping, where each transformer block learns to iterate its hidden state via a learned halting mechanism, and gated memory banks, that provide additional learned storage. We find that looping primarily benefits mathematical reasoning, while memory banks help recover performance on commonsense tasks compared to parameter and FLOP matched models. Combining both mechanisms yields a model that outperforms an iso-FLOP baseline—with three times the number of layers—on math benchmarks. Analysis of model internals reveals layer specialization: early layers learn to loop minimally and access memory sparingly, while later layers do both more heavily. 1 Introduction Large language models can reason explicitly via chain-of-thought (CoT) prompting ( 22 ) , which uses step-by-step verbalization to produce reasoning traces, improving performance in downstream tasks. While this is effective, each reasoning step requires generating tokens ( 14 ) , which has motivated interest in implicit reasoning, where models perform multi-step computation within their hidden representations without producing intermediate text ( 18 ; 1 ) . One way of implementing implicit reasoning is by stacking transformer layers, iteratively applying the same block which refines the representations through repeated computation. This makes efficient use of parameters—a model that loops N times achieves a larger effective depth without using N times the parameters ( 9 ; 4 ; 2 ; 7 ) . Recent work has shown that looped transformers can match much deeper non-looped models on reasoning tasks ( 18 ; 24 ; 16 ) . However, a looped model has fundamentally less capacity than a deeper model with N times the number of layers. While loops may improve reasoning, the model has fewer unique parameters in which to encode knowledge. Recent analysis suggests this trade-off is fundamental: looped models achieve their parameter efficiency not through increased knowledge storage but through knowledge manipulation —they are able to do multi-hop reasoning while showing similar per-parameter memorization capacity to standard transformers ( 24 ) . Here, we investigate whether learned memory banks can restore the missing capacity. Specifically, we make the following contributions: (1) we propose an adapted looped Transformer that combines per-layer adaptive looping with gated access to local and global memory, and (2) we conduct a systematic study examining the effects of adaptive looping and the inclusion of memory banks on downstream model performance. We find that looping primarily benefits mathematical reasoning, while memory banks help recover commonsense performance compared to parameter- and FLOP-matched models. Analysis of model internals reveals layer specialization: early layers learn to loop minimally and access memory sparingly, while later layers do both more heavily. This specialization means the model learns to choose between thinking harder and knowing more and where to do each. Base Model (Transformer) \mathbf{h}^{(0)} Block 1 \mathbf{h}^{(1)} Block 2 \mathbf{h}^{(2)} \vdots Block L \mathbf{h}^{(L)} Loop Model (Adaptive Computation) \mathbf{h}^{(\ell-1)} Block \ell \times\zeta(s_{n}) \times N \sum weighted \mathbf{h}^{(\ell)} p_{1} p_{2} \vdots p_{N} halt probs Memory Model (Adaptive + Memory) Global Mem (shared) Local Mem (layer \ell ) \mathbf{h}^{(\ell-1)} + Block \ell \times\zeta(s_{n}) \sigma_{G} \sigma_{L} \times N \sum \mathbf{h}^{(\ell)} p_{1} p_{2} \vdots p_{N} halt probs Figure 1: Architecture overview. Left: A standard transformer passes hidden states through L unique blocks. Center: Our loop model allows each block to iterate up to N times, with a learned halting mechanism that produces a weighted combination of intermediate states. Per-step scales \zeta(s_{n}) are initialized near zero for training stability. Right: The combined model additionally retrieves from local (per-layer) and global (shared) memory banks, gated by learned input-dependent scalars. 2 Methods We augment a standard decoder-only transformer ( 21 ) with two mechanisms: adaptive looping for repeating computation and memory banks for retrieving learned knowledge. Figure 1 illustrates the architecture and Appendix A.1 provides additional details. 2.1 Adaptive Looping A standard transformer block applies multi-head self-attention followed by a feed-forward network using residual connections and layer normalization: \displaystyle\mathbf{h}^{\prime} \displaystyle=\mathbf{h}+\text{Attn}(\text{LN}(\mathbf{h})) (1) \displaystyle\mathbf{h}^{\prime\prime} \displaystyle=\mathbf{h}^{\prime}+\text{FFN}(\text{LN}(\mathbf{h}^{\prime})) (2) where \mathbf{h}\in\mathbb{R}^{B\times T\times D} is the hidden state with batch size B , sequence length T , and embedding dimension D . We allow each transformer block to be applied multiple times with a learned halting mechanism, inspired by PonderNet ( 2 ) . At each iteration t\in\{1,\ldots,N_{\max}\} , a halting router predicts the probability of stopping: p_{t}=\sigma\left(\mathbf{W}_{h}\left[\mathbf{h}^{(t)};t/N_{\max}\right]+b_{h}\right) (3) where [\cdot;\cdot] denotes concatenation and t/N_{\max} provides a normalized step embedding. The final output is computed as a weighted combination over all iterations: \mathbf{h}_{\text{out}}=\sum_{t=1}^{N_{\max}}p_{\text{halt}}^{(t)}\cdot\mathbf{h}^{(t)} (4) where p_{\text{halt}}^{(t)}=p_{t}\prod_{i=1}^{t-1}(1-p_{i}) is the probability of halting at exactly step t . Learnable Loop Scales. To stabilize model training, we introduce per-step learnable scale parameters. Each iteration applies: \mathbf{h}^{(t)}=\mathbf{h}^{(t-1)}+\text{softplus}(\alpha_{t})\cdot f_{\theta}(\text{LN}(\mathbf{h}^{(t-1)})) (5) where f_{\theta} denotes the transformer block and \alpha_{t} is initialized to -7.0 , which ensures the loop begins as an approximate identity mapping, and the model gradually learns when and how much to intervene. 2.2 Memory Banks We introduce two types of learned memory, a local and a global one. For the Local (Per-Layer) Memory each layer \ell maintains its own memory bank (\mathbf{K}_{\ell},\mathbf{V}_{\ell})\in\mathbb{R}^{M_{L}\times D} with M_{L} slots. This enables layer-specific storage of intermediate computations or specialized knowledge appropriate to that depth. The Global (Shared) Memory uses a single memory bank (\mathbf{K}_{G},\mathbf{V}_{G})\in\mathbb{R}^{M_{G}\times D} that is shared across all layers, allowing storage of information that might be beneficial for all layers. Memory retrieval uses scaled dot-product attention with QK-normalization ( 3 ) : \displaystyle\mathbf{m}_{\text{local}} \displaystyle=\text{softmax}\!\left(\frac{\text{LN}_{q}(\mathbf{h})\cdot\text{LN}_{k}(\mathbf{K}_{\ell})^{\top}}{\sqrt{D}}\right)\mathbf{V}_{\ell} (6) \displaystyle\mathbf{m}_{\text{global}} \displaystyle=\text{softmax}\!\left(\frac{\text{LN}_{q}(\mathbf{h})\cdot\text{LN}_{k}(\mathbf{K}_{G})^{\top}}{\sqrt{D}}\right)\mathbf{V}_{G} (7) Unlike the KV-cache in standard attention, which stores activation history during inference, our memory banks are static learnable parameters that are optimized via backpropagation during training but fixed during inference. Our memory implementation draws inspiration from memory-augmented architectures ( 11 ; 19 ; 23 ) and neural Turing machines ( 8 ) . Gated Memory Integration A critical design choice is how to integrate retrieved memory into the residual stream. Naive addition would force the model to always use memory, potentially harming performance on tasks where loops alone suffice. We therefore employ input-dependent gating: \displaystyle\mathbf{g} \displaystyle=\sigma(\mathbf{W}_{g}\mathbf{h}+b_{g}) (8) \displaystyle\mathbf{h}_{\text{enriched}} \displaystyle=\mathbf{h}+\mathbf{g}\odot\mathbf{W}_{m}\mathbf{m} (9) where \odot denotes element-wise multiplication. Separate gates control local and global memory contributions: \mathbf{h}_{\text{memory}}=\mathbf{h}+\mathbf{g}_{L}\odot\mathbf{W}_{L}\mathbf{m}_{\text{local}}+\mathbf{g}_{G}\odot\mathbf{W}_{G}\mathbf{m}_{\text{global}} (10) We study the effect of gate bias initialization b_{g} , comparing b_{g}\in\{-3,0,3\} corresponding to initial gate activations of approximately \sigma(-3)\approx 0.05 (nearly closed), \sigma(0)=0.5 (balanced) and \sigma(3)\approx 0.95 (nearly open). 3 Results 3.1 Experimental Setup Model. Our base architecture is a decoder-only transformer with L=12 layers and a total of {\sim} 200M parameters (see Appendix A.1 for full details). For looped models, we use the same 12-layer architecture and allow each layer to iterate up to N_{\max}\in\{3,5,7\} times. For memory-augmented models, we add M_{L}= 1024 local memory slots per layer and M_{G}= 512 global memory slots, which in total adds approximately 10M parameters. We adapt our iso-param and