HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning

Long H Dang, David Rawlinson · 2025

Read HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning by Long H Dang, David Rawlinson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22832.txt.

Introduction

A small reasoning model may adapt its effort to task difficulty. Long H Dang and David Rawlinson extend that model in 2025. The base model is the Hierarchical Reasoning Model. It had only faced static, supervised tasks. Real tasks are dynamic and partly hidden.

Their variant is called HRM-Agent. It trains with reinforcement learning only. Reinforcement learning learns from rewards, not labels. The agent navigates mazes that change over time. Goals must be reached under uncertainty. The team probes its recurrent inference process. Evidence suggests it reuses past computation. That reuse spans earlier time steps. Such reuse suits real world sequential problems.

Catalog record

CREATORLong H Dang, David Rawlinson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22832

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning Long H Dang, David Rawlinson Abstract The Hierarchical Reasoning Model (HRM) has impressive reasoning abilities given its small size, but has only been applied to supervised, static, fully-observable problems. One of HRM’s strengths is its ability to adapt its computational effort to the difficulty of the problem. However, in its current form it cannot integrate and reuse computation from previous time-steps if the problem is dynamic, uncertain or partially observable, or be applied where the “correct” action is undefined, characteristics of many real-world problems. This paper presents HRM-Agent, a variant of HRM trained using only reinforcement learning. We show that HRM can learn to navigate to goals in dynamic and uncertain maze environments. Recent work suggests that HRM’s reasoning abilities stem from its recurrent inference process. We explore the dynamics of the recurrent inference process and find evidence that it is successfully reusing computation from earlier environment time-steps. 1 Introduction 1.1 Reasoning models Most Large Language Models (LLMs) sequentially generate a stream of output tokens as part of a forward inference process of pre-determined computational complexity. The inference process does not intrinsically adapt to the complexity of the task, leading to inefficiency. Unreliable performance is often reported, especially in tasks (including maze navigation) where the model must jointly review all its outputs to validate an appropriate response ( Jolicoeur-Martineau 2025 ) . Several techniques have been proposed to remedy this weakness, including “Chain of Thought” (CoT) ( Wei et al. 2022 ) and Tree-of-Thought (ToT) exploration ( Yao et al. 2023 ) , in which the model explicitly generates intermediate steps before outputting a final decision; “Thinking tokens” ( Herel and Mikolov 2024 ) , which provide additional inference steps; scaling test-time compute ( Snell et al. 2024 ) , and “agentic” architectures such as the “AI Scientist” ( Lu et al. 2024 ) . However, all these strategies ultimately exploit human insight and ingenuity to devise ways to coax better results from broadly unchanged models. Most current agentic systems require human thought to design a process to decompose broad problems into smaller sub-problems the model will likely find more manageable, or where it will perform more consistently. This seems inadvisable given that one of the key lessons from deep learning is that learned models tend to outperform human-defined problem decompositions ( LeCun et al. 2015 ) . Wang et al. 2025 recently proposed the Hierarchical Hierarchical Reasoning Model (HRM) which demonstrated data-efficient reasoning in a relatively compact model. Importantly, given the complaint about LLMs above, HRM has a recurrent inference process which can adapt to the complexity of the current task. The HRM paper includes a demonstration of automatic, inference-time scaling of computational complexity using the Adaptive Computation Time (ACT) component, which determines the number of recurrent iterations to perform. HRM employs dual recurrent modules - a “high-level”, slow-updating module H , and a “low-level”, fast-updating module L . The different levels of abstraction in H and L are intended to emerge from convergence dynamics and are not explicitly enforced. Both models work together, initially in a recurrent mode to convergence, and then in a single forward pass without explicit supervision of intermediate steps to produce a complete solution to a fully-observable, static problem. In their paper, this architecture is shown to solve complex reasoning tasks such as extreme Sudoku, maze path planning, and tasks from the Abstraction and Reasoning Corpus (ARC) ( Chollet 2019 ) with orders of magnitude fewer parameters ( \sim 27 M) and training examples ( \sim 1 k) than traditional deep reasoning models. HRM’s reported combination of performance and efficiency is interesting because much larger reasoning models appear to plateau in performance despite scaling the available computational resources ( Shojaee et al. 2025 ) , suggesting a fundamental limitation. Since publication of the HRM paper, other authors have explored derivative models. In a blog post by the ARC-prize foundation ( ARC-Prize-team 2025 ) it was reported that the largest contributor to model performance was the recurrent process. More recently, Jolicoeur-Martineau 2025 published the Tiny Recursive Model (TRM). Via a series of ablations and variants, the TRM paper reports that the recurrent (recursive) forward process is key to its reasoning abilities. The TRM combines the H and L modules but retains the alternating pivot between two latent vectors during recurrent convergence. 1.2 Reasoning in dynamic, uncertain or partially observable environments If HRM can unlock improved reasoning abilities in static and fully observable tasks, can these benefits be realized in dynamic, uncertain or partially observable environments? This paper aims to explore whether the reasoning abilities of HRM can be used in the context of an agent which is trained using only reinforcement learning, without supervised losses. This would enable HRM to be applied to problems which cannot be tackled by simply by analysis of the initial problem state. In these problems, an agent must incrementally develop a solution via a series of actions, and continually adapt its solution to changes in the environment. Reinforcement learning (RL) has been successfully applied in various domains, from Atari gameplay ( Mnih et al. 2015 ) to robotic control ( Kober et al. 2013 ) . RL is appropriate when it is impractical or simply not meaningful to define a correct response. Instead, models maximize qualitative feedback as given as rewards. Rewards may be sparse, meaning that most states provide no feedback at all. This makes RL suitable for tasks which require incremental solution construction. “Reasoning” in a RL context can be expressed in many ways, including long horizon path planning ( Sutton et al. 1999 ) , multi-step instruction following ( Chevalier-Boisvert et al. 2023 ; Chevalier-Boisvert et al. 2018 ) , logical reasoning about entities ( Zambaldi et al. 2019 ) , discovery of causal, hierarchical sequences (e.g. making and using tools) ( Hafner 2021 ) , arithmetic reasoning ( Saxton et al. 2019 ) , theory of mind ( Bard et al. 2020 ) , multi-agent planning ( Vinyals et al. 2017 ) and abstract pattern generalization ( Cobbe et al. 2019 ) . In this paper we have focused on the first of these, path planning. Many real-world RL problems are dynamic, uncertain and partially observable (PO). For example, a mobile robot must move around to sense different locations in the environment. Partially Observable Markov Decision Processes (POMDPs) ( Kaelbling et al. 1998 ) capture settings where the agent cannot directly observe the full environment state, requiring it to maintain an internal belief over possible states. In dynamic and uncertain environments, where transition dynamics and relevant observations may change unpredictably, agents must continually integrate new information, while discarding outdated knowledge. Solution construction is usually incremental and cannot be determined in a single observation–action cycle regardless of the initial reasoning effort or quality. RL problems can be fully-observable but still dynamic and uncertain. In a Markov Game such as Chess or Go the state of the board is observable but the actions of the opposing player are unpredictable. AlphaGo ( Silver et al. 2016 ) used deep reinforcement learning in combination with Monte Carlo Tree Search to demonstrate reasoning and planning over long time horizons.

Read the complete text