When Can Model-Free Reinforcement Learning be Enough for Thinking?

Josiah P. Hanna, Nicholas E. Corrado · 2025

Read When Can Model-Free Reinforcement Learning be Enough for Thinking? by Josiah P. Hanna, Nicholas E. Corrado (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17124.txt.

Introduction

Josiah Hanna and Nicholas Corrado ask when plain trial-and-error learning, with no world model, can produce behavior that looks like thinking. Their paper appeared in 2025. Language models trained this way have begun to reason aloud, which is odd: thought steps earn no reward and change nothing outside. The team builds a minimal theory by adding a thought state and thought actions to the classic decision model.

They prove the starting policy decides whether thinking emerges at all, and that a thought step equals one round of self-improvement of the policy. Open language models meet the conditions their theory requires. They then propose what might let thinking arise beyond language, with a toy test case.

Catalog record

CREATORJosiah P. Hanna, Nicholas E. Corrado
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17124

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Can Model-Free Reinforcement Learning be Enough for Thinking? Josiah P. Hanna, Nicholas E. Corrado Abstract Recent work on large language models has demonstrated the use of model-free reinforcement learning (RL) to train reasoning-like capabilities. The emergence of "thinking" through model-free RL is interesting as thinking actions neither produce reward nor change the external world state to one where the agent is more likely to get reward. This paper seeks to build a domain-independent understanding of when model-free RL will lead to "thinking" as a strategy for reward maximization. To build this understanding, we first introduce a theoretical model which we call a thought Markov decision process (MDP). Thought MDPs minimally extend the classical MDP model to include an abstract notion of thought state and thought action. Using the thought MDP model, we prove the importance of policy initialization in determining whether or not thinking emerges and show formally that thought actions are equivalent to the agent choosing to perform a step of policy improvement before continuing to act. We then show that open-source LLMs satisfy the conditions that our theory predicts are necessary for model-free RL to produce thinking-like behavior. Finally, we hypothesize sufficient conditions that would enable thinking to be learned outside of language generation and introduce a toy domain where a combination of multi-task pre-training and designated thought actions enable more data-efficient RL compared to non-thinking agents. 1 Introduction Dual process theory divides cognitive processing into System 1 and System 2 processing ( Wason and Evans 1974 ; Kahneman 2003 ) . System 1 processing is fast, effortless, but potentially imprecise, whereas System 2 is slow, effortful, but potentially produces better decisions. In the field of computational reinforcement learning (RL), model-free learning is reminiscent of System 1 behavior, whereas model-based decision-time planning is reminiscent of System 2. While dual process theory is a popular model for understanding human cognition ( Kahneman 2011 ) , there is no single clear candidate architecture for its duplication in artificial agents. In the context of large language models (LLMs), recent work by Guo et al. 2025 proposed a unified model whereby System 2 type “thinking” emerged as a consequence of model-free RL applied to solve mathematics problems ( Guo et al. 2025 ). While this “thinking" appears to resemble thoughts that a human would have when answering a given query, these outputs arise solely as subservient to reward maximization. The approach is interesting as it suggests that a form of System 2 processing can emerge when we view thinking as control and reinforce thought patterns that lead to reward. Our objective is to develop a domain-independent understanding of the conditions under which model-free RL will select for thinking behavior. Specifically, we want to answer the question: Under what conditions will model-free reinforcement learning give rise to thinking as a strategy for reward maximization? Informally, we define thinking to be actions that do not directly produce reward or affect the external state of an agent’s environment but that lead the agent to take a course of action that increases the reward it will receive in the future. To answer our research question, we first formulate a minimal extension of the classical MDP model to explicitly model thought actions as well as a notion of a controllable thought state. We then show how policy initialization plays a central role in whether policy iteration will select thought actions. Under our theoretical model, we will show that thinking can be viewed as selecting between a set of sub-policies that are already contained in the learning agent’s policy function and that thought actions can be interpreted as the agent choosing to run one or more steps of policy improvement before continuing to act. We then discuss how LLMs instantiate our thought MDP formalism and provide empirical evidence that they exhibit the necessary conditions for thinking to emerge. Our final contribution is to introduce a simple domain and multi-task pre-training set-up that induces the condition under which model-free RL will discover thinking behavior. This simple domain provides a basis for future work studying agents that learn to think and act. We conclude by discussing open questions and directions for future work raised by this model of deliberative thinking in RL. 2 Related Work In this section, we discuss the prior literature related to System 2 processing in RL as well as prior models of “mental" actions an agent can take. Decision-time Planning A hallmark of System 2 processing is deliberative planning as opposed to reflexive action. Thus, a natural way to instantiate System 2 in artificial agents is to use decision-time planning, e.g., Monte Carlo tree search ( Kocsis and Szepesvári 2006 ) , and many works in the RL literature have used this approach ( Anthony et al. 2017 ; Silver et al. 2016 ; Silver et al. 2017 ; Silver 2009 ; Schrittwieser et al. 2021 ; Shah et al. 2020 , e.g.,) . Decision-time planning is particularly effective when the agent’s policy or value function is sub-optimal but the agent possesses an accurate model of state transitions ( Sutton and Barto 2018 ) . In this case, planning can be viewed as focused policy improvement to improve the decision at the agent’s current state before it acts. As we will show, the idea of thinking as local policy improvement also applies when an RL agent learns to think in a thought MDP. The key difference is that we consider a more abstract model of thinking without any notion of forward prediction and search. There is also no natural answer to the question of when to stop planning and act with most decision-time planning methods. MCTS is an anytime algorithm and the general practice is to allow as much search is possible in the application domain ( Silver et al. 2016 ; Silver et al. 2017 ) . On the other hand, model-free learning in our thought MDP model will directly relate the decision about when to stop thinking to the objective of return maximization. Some work in the (non-learning) planning literature has also studied the question of when to think vs. when to act ( Cashmore et al. 2019 ; Coles et al. 2024 ) . Learning to Plan As neural networks are general-purpose function approximators, they can, in principle, learn algorithmic planning or reasoning computations and prior work has attempted to induce planning capabilities through specialized architectures ( Tamar et al. 2017 ; Farquhar et al. 2018 ; Guez et al. 2019 ; Guez et al. 2018 ; Niu et al. 2018 ; Weber et al. 2018 ; Sykora et al. 2020 ; Schleich et al. 2019 ; Oh et al. 2017 ) . Guez et al. 2019 showed that planning-like computations could emerge through model-free learning ( Guez et al. 2019 ). Bush et al. 2025 applied mechanistic interpretability approaches to show that the approach of Guez et al. 2019 did in fact learn planning computations (i.e., the network internally learned to propose and evaluate future candidate action sequences). While such works also show that model-free RL can produce thinking behavior, the main difference is that thinking refers to the computation done by the agent’s policy, whereas we study thinking as an

Read the complete text