Unlocking the Working Memory of Large Language Models for Latent Reasoning

Lukas Aichberger Sepp Hochreiter · 2026

Read Unlocking the Working Memory of Large Language Models for Latent Reasoning by Lukas Aichberger Sepp Hochreiter (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30343.txt.

Introduction

Lukas Aichberger and Sepp Hochreiter introduce Reasoning in Memory (RiM), a latent method replacing autoregressive reasoning steps with memory blocks. Human cognition uses working memory to hold and handle information inside without sharing thoughts. RiM draws on this principle with fixed sets of tokens opening working memory. They employ a two-stage plan: first grounding memory blocks by predicting reasoning steps after each block, then dropping step checks and tuning the final answer step by step. Tests show RiM matches or exceeds latent methods while skipping autoregressive thoughts.

Catalog record

CREATORLukas Aichberger Sepp Hochreiter
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66728
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30343

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unlocking the Working Memory of Large Language Models for Latent Reasoning Lukas Aichberger Sepp Hochreiter Abstract To improve the reasoning capabilities of large language models, test-time compute is typically scaled by generating intermediate tokens before the final answer. However, this couples reasoning to autoregressive generation and thereby conflates internal computation with external communication. In contrast, human cognition can use working memory to hold and manipulate information internally without the need to externalize intermediate thoughts. Drawing on this principle, we introduce Reasoning in Memory (RiM) , a latent reasoning method that replaces the autoregressive generation of reasoning steps with memory blocks. These memory blocks are fixed sequences of special tokens that unlock the working-memory capacity of large language models. Since they are fixed rather than generated, they can be processed in a single forward pass, enabling compute-efficient latent reasoning. To operationalize these memory blocks, we employ a two-stage curriculum. First, we ground them by predicting explicit reasoning steps after each memory block. Second, we discard this step-level supervision and iteratively refine the final answer after each memory block. Our experiments on reasoning benchmarks show that, across language models of different families and sizes, RiM matches or exceeds existing latent reasoning methods while avoiding the autoregressive generation of thoughts. These results demonstrate that large language models can be trained to use working memory as an effective mechanism for latent reasoning. 1 Introduction Figure 1 : Reasoning in Memory (RiM). Stage 1 trains the LLM to use memory blocks (yellow) as working memory by supervising the prediction of the next reasoning step (blue) after each memory block. Once the memory blocks are grounded for intermediate computation, Stage 2 removes reasoning-step supervision and trains the LLM to refine the final answer after each memory block. Large language models (LLMs) have demonstrated remarkable reasoning capabilities, primarily driven by techniques that scale test-time compute through generating intermediate tokens before giving the final answer ( Wei et al. 2022 ; Snell et al. 2024 ) . In chain-of-thought (CoT) reasoning, intermediate computation remains directly coupled to the generation of text. This coupling forces an LLM to “think out loud”, anchoring its reasoning process to the syntax and structure of natural language. However, language is optimized for communication rather than for computation ( Wei et al. 2022 ; Kojima et al. 2022 ) . Consequently, part of the computational budget is allocated for generating grammatical and fluent intermediate text, rather than being devoted purely to internal computation. Recent advances in latent reasoning bypass these natural-language constraints by replacing discrete tokens with continuous representations ( Hao et al. 2025 ; Shen et al. 2025 ; Cheng and Durme 2024 ) . However, they preserve the same step-by-step generation paradigm, effectively “thinking out loud” in a continuous space. Whether generating discrete tokens or continuous representations, each intermediate computation must still be externalized before future computation can condition on it. Thus, these latent reasoning methods preserve the coupling of intermediate computation to autoregressive generation. Human cognition suggests a different design principle. When solving complex problems, humans do not typically articulate every intermediate step, nor is their reasoning strictly bound to the syntax of language. Instead, cognitive psychology identifies working memory as an internal workspace for holding and manipulating task-relevant representations ( Baddeley 1992 ) . Developmental psychology observes that while we initially rely on “thinking out loud”, this linguistic scaffolding is eventually abandoned in favor of internal processing ( Vygotsky 1978 ) . This motivates a shift from generating intermediate reasoning steps autoregressively to representing them in a dedicated internal workspace of an LLM, analogous to human working memory. Drawing on these cognitive principles, we introduce Reasoning in Memory (RiM) , which operationalizes memory blocks , fixed sequences of special tokens, as a working memory for latent reasoning. Rather than generating intermediate reasoning steps autoregressively, our method trains the LLM to use these memory blocks as an internal latent workspace for task-relevant computation. While their token identities and positions are fixed, their contextual representations remain task-dependent and can encode intermediate reasoning. A direct benefit of these fixed memory blocks is that they can be processed in a single forward pass. This avoids the bottleneck of autoregressively generating discrete tokens or continuous representations, making our method highly parallelizable and compute-efficient. Figure 2 : RiM Attention Mask. Memory blocks (yellow) attend to the question and previous memory blocks. Target reasoning steps (blue) attend to previous memory blocks and optionally the question, but not to other reasoning steps. This enables all targets to be predicted in one forward pass without information leakage, forcing reasoning inside the memory blocks. In order to teach the LLM to use these memory blocks as a latent workspace, RiM employs a two-stage curriculum ( Figure 1 ). Since the memory blocks have no predefined computational role, unlocking their working-memory capacity requires a carefully structured training signal. In Stage 1, we therefore ground the memory blocks by supervising the readout after each block to predict the corresponding next reasoning step. As each readout can recover this target only from memory blocks seen so far, the LLM is forced to structure the latent workspace around task-relevant intermediate information. In Stage 2, once the working-memory capacity is learned, we remove this strict step-level supervision and instead supervise the readout after each block to predict the final answer. The training signal is therefore shifted from predicting written reasoning steps to progressively refining the answer across memory blocks. A custom attention mask ( Figure 2 ) allows all readouts to be trained in a single forward pass, making supervision dense and efficient in both stages (see Appendix D for details). Empirically, we validate RiM by training language models across GPT-2 and Llama-3.2 scales on GSM8K-Aug and evaluating them on GSM8K and GSM-Hard as in- and out-of-distribution benchmarks, respectively. Across these settings, RiM matches or exceeds existing CoT and latent reasoning baselines. These findings show that our method enables LLMs to use working memory for efficient reasoning by combining dense supervision during training with compute-efficient inference. Our main contributions are as follows: • We introduce Reasoning in Memory (RiM), a latent reasoning method that unlocks the working-memory capacity of LLMs by providing a latent workspace for intermediate computation. • We propose a two-stage curriculum that assigns memory blocks a computational role through carefully structured training signals, enabling their use as working memory for latent reasoning. • We show that RiM matches or exceeds explicit and latent reasoning baselines across model scales and reasoning benchmarks, while making intermediate computation parallelizable and efficient. 2 Related Work CoT reasoning. The modern test-time-compute paradigm grew out of methods that place a textual workspace between the question and the answer ( Nye et al. 2021 ) . Chain-of-thought (CoT) prompting ( Wei et al. 2022 ; Kojima et al. 2022 ) showed that language models solve harder tasks when intermediate computation is externalized as text. First, explicit reasoning has been scaled at inference time by repeated sampling ( Brown et al. 2024 ) , aggregating multiple reasoning traces ( Wang et al. 2023 ) , structured search ( Yao et al. 2023 ) , or process reward models that verify intermediate steps ( Lightman et al. 2024 ; Snell et

Read the complete text