Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves

Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting · 2026

Read Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves by Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21582.txt.

Introduction

Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, and Kristian Kersting give latent reasoning in language models the attention it lacks. Depth recurrence lets a model think by looping through shared layers. Past work, though, compared unfair baselines and left much of the depth unused.

A fixed hidden size also chokes long chains of hidden thought. The authors mix attention across depths to loosen that choke point. Their checks match rivals on compute, size, and memory for fairness. Stronger hidden reasoning could let small models punch above their weight on hard problems without writing long visible chains.

Catalog record

CREATORJonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21582

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting Abstract Depth-recurrence facilitates latent reasoning by sharing parameters across depths. However, prior work lacks combined FLOP-, parameter-, and memory-matched baselines, underutilizes depth-recurrence due to partially fixed layer stacks, and ignores the bottleneck of constant hidden-sizes that restricts many-step latent reasoning. To address this, we introduce a modular framework of depth-recurrent attention mixtures (Dreamer), combining sequence attention, depth attention, and sparse expert attention. It alleviates the hidden-size bottleneck through attention along depth, decouples scaling dimensions, and allows depth-recurrent models to scale efficiently and effectively. Across language reasoning benchmarks, our models require 2 to 8 × \times fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched SOTA, and outperform ca. 2 × \times larger SOTA models with the same training tokens. We further present insights into knowledge usage across depths, e.g., showing 2 to 11 × \times larger expert selection diversity than SOTA MoEs. 1 Introduction Reasoning language models are becoming the backbone of modern AI for various applications. They allow tackling complex problems by scaling test-time compute via chain-of-thought (CoT). But traditional CoT enforces verbalization in discrete natural language, which limits expressivity and results in long sequences that are computationally expensive to generate and train. A potential solution is latent reasoning via depth recurrence (DR). By reusing parameters across depths, DR allows scaling the depth while keeping the model and its parameter count unchanged. This allows open-ended many-step reasoning in latent space along the depth dimension, offering an alternative to traditional CoT. “Latent CoT” in continuous space can increase reasoning efficiency, leveraging expressivity beyond natural language ( 47 ; 14 ) . Additionally, knowledge reuse across depths may improve parameter efficiency and compositional generalization, a crucial property for out-of-distribution performance. However, there are two major bottlenecks for scaling DR: Layer-size bottleneck : Scaling a dense DR model is intractable, since the whole DR model is executed at each depth. To solve this, we decouple compute from parameters by using a sparse mixture of experts (MoE), with adjustements like low-rank and depth-position-encoded routing, to improve behavior in depth-recurrent settings. Hidden-size bottleneck : Pushing more reasoning from the sequence into the depth dimension exacerbates the hidden-size bottleneck, which limits complex latent reasoning due to a constant hidden-state size. Attention already solves the same issue observed in recurrent neural networks (RNNs) along the sequence dimension, leading to sequence attention (SA). We suggest to leverage the same principle along the depth dimension, thus leading to depth attention (DA). Figure 1 : High-level illustration of our modular d epth- re current a ttention m ixtu re ( Dreamer ). This instance combines sequence attention (SA), depth attention (DA), and expert attention (EA) in a single depth-recurrent (DR) layer. It facilitates knowledge reuse , compositional generalization , and alleviates the hidden-size bottleneck , leading to better reasoning , data efficiency , and scaling behavior . The decoupled scaling dimensions (#params, #FLOPs, depth, and latent memory size) are individually adjustable. After formulating MoEs as expert attention (EA), we arrive at a unified view of SA, DA, and EA as an attention mixture over sequence, depth, and experts. This provides a unification of knowledge access along all major dimensions via attention. Such a homogenous architecture may help to better understand and control model behavior, and facilitates modular extensibility via additional attention dimensions. In summary, our main contributions are as follows: 1. Methods: We introduce depth-recurrent attention mixtures, a modular, unifying framework that combines sequence attention, depth attention, and sparse expert attention. It adapts sparse MoEs for depth recurrence and alleviates the hidden-size bottleneck, allowing depth-recurrent models to scale efficiently and effectively. 2. Experiments: We perform tightly FLOP-, parameter-, and memory-matched comparisons between our methods and SOTA MoEs, for isolated ablations of depth recurrence and depth attention. We show consistent, large accuracy gains for natural language reasoning. 3. Analysis: We provide insights into knowledge usage across depths, from the perspectives of depth attention and expert attention. In high resolution (with over thousand small experts), we quantify knowledge capacity allocation and reuse patterns across depths, and derive practical suggestions for model design. 2 Related Work Depth Recurrence and Parameter Sharing across Depth: Depth-recurrent (DR) Transformers are introduced by 10 While laying the groundwork, they miss FLOP-matched tests and use a dense DR core, making scaling intractable. To address the scalability bottleneck of dense DR, 36 investigates DR with sparse MoEs. However, they still lack FLOP-matching and testing of natural language reasoning. 8 perform more rigorous and scaled experiments with sparse DR, showing slight benefits of DR. However, they lack sparse baselines for fair comparisons. Moreover, they rely on multi-layer DR cores, in contrast to our minimal, modular single-layer architecture. 4 ; 37 ; 24 investigates the reuse of MoE experts across (some adjacent) depths, thus resembling a form of partial/block-wise DR with MoEs. However, the partial DR prevents depth-generalized and open-ended latent reasoning. The same limitation applies to work of 3 , exploring DR with depth-specific LoRA-adapters as relaxation. In 2025, we saw a surge of work on DR, for natural language reasoning ( 33 ; 12 ; 23 ; 42 ; 48 ; 28 ) and small models for symbolic problems ( 9 ; 40 ; 21 ) . Despite the intractability of scaling dense DR, most of these works use dense models. Moreover, as proposed by 12 , many of them, especially for natural language, use dedicated encoders/decoders around the DR core. In contrast, we show that purely a single-layer DR core is sufficient. It can learn such dedicated roles of encoding/decoding on its own if useful, while maintaining a homogeneous architecture, maximizing knowledge reuse and facilitating interpretability and modular extensibility. Further, it allows batching of inference steps in different depths, e.g. for speculative decoding and/or batched sequences with different/dynamic depths. In addition, all mentioned works on DR so far ignore the hidden-size bottleneck, which becomes more pronounced as depth increases and more reasoning shifts from the sequence into the depth dimension. In contrast, our work addresses both hidden-size bottleneck and dense layer-size bottleneck. Other Latent Reasoning Approaches: 20 decouples input and latent reasoning dimensions by cross-attending to the input sequence from a latent, recurrent dimension. However, training is not naively parallelizable across the output sequence. As another alternative, 14 generates continuous latent tokens between discrete I/O tokens in the sequence dimension. However, this neither supports naive sequence-parallel training nor leverages knowledge-reuse across depths. Moreover, latent steps extend the sequence length, thus increase memory and compute requirements. In contrast, our use of DR with depth attention operates on the depth dimension, overwriting keys/values of depth attention after each token, thus incurring only negligible compute and memory overhead, while still supporting sequence-parallel training. Depth Attention and Skip Connections: As precursor to depth attention (DA), 19 explores skip connections between all layers, although only for shallow convolution networks for vision. A different approach by 32 averages hidden-states from previous layers, weighted by fixed learned weights. 43 computes dynamic weights for such an aggregation via an MLP. All these methods still do not resemble proper dot-product attention along the depth. In contrast, 11 and 6 do use such

Read the complete text