Unrewarded Exploration in Large Language Models Reveals Latent Learning from Psychology

Jian Xiong, Jingbo Zhou, Zihan Zhou, Yixiong Xiao, Le Zhang, Jingyong Ye, Rui Qian, Yang Zhou, Dejing Dou · 2026

Read Unrewarded Exploration in Large Language Models Reveals Latent Learning from Psychology by Jian Xiong, Jingbo Zhou, Zihan Zhou, Yixiong Xiao, Le Zhang, Jingyong Ye, Rui Qian, Yang Zhou, Dejing Dou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22474.txt.

Introduction

Jian Xiong and a large team find hints of latent learning — grasp gained with no reward — inside large language models. Tolman's rats learned a maze while just wandering, then used that grasp once food appeared. Their 2026 preprint contrasts this with reward-led learning, which leans hard on outside feedback.

The team probe whether models build inner maps during unrewarded play. If they do, those maps could let models adapt fast when goals arrive. The claim links model science to a classic idea from psychology. It may reshape how agents are trained to explore.

Catalog record

CREATORJian Xiong, Jingbo Zhou, Zihan Zhou, Yixiong Xiao, Le Zhang, Jingyong Ye, Rui Qian, Yang Zhou, Dejing Dou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22474

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unrewarded Exploration in Large Language Models Reveals Latent Learning from Psychology Jian Xiong, Jingbo Zhou, Zihan Zhou, Yixiong Xiao, Le Zhang, Jingyong Ye, Rui Qian, Yang Zhou, Dejing Dou Abstract Latent learning, classically theorized by Tolman, shows that biological agents (e.g., rats) can acquire internal representations of their environment without rewards, enabling rapid adaptation once rewards are introduced. In contrast, from a cognitive science perspective, reward learning remains overly dependent on external feedback, limiting flexibility and generalization. Although recent advances in the reasoning capabilities of large language models (LLMs), such as OpenAI-o1 and DeepSeek-R1, mark a significant breakthrough, these models still rely primarily on reward-centric reinforcement learning paradigms. Whether and how the well-established phenomenon of latent learning in psychology can inform or emerge within LLMs’ training remains largely unexplored. In this work, we present novel findings from our experiments that LLMs also exhibit the latent learning dynamics. During an initial phase of unrewarded exploration, LLMs display modest performance improvements, as this phase allows LLMs to organize task-relevant knowledge without being constrained by reward-driven biases, and performance is further enhanced once rewards are introduced. LLMs post-trained under this two-stage exploration regime ultimately achieve higher competence than those post-trained with reward-based reinforcement learning throughout. Beyond these empirical observations, we also provide theoretical analyses for our experiments explaining why unrewarded exploration yields performance gains, offering a mechanistic account of these dynamics. Specifically, we conducted extensive experiments across multiple model families and diverse task domains to establish the existence of the latent learning dynamics in LLMs. † † affiliationnotice: * Work done when intern at Baidu Research 1 Introduction Figure 1 : Overview of tasks and models we use in this work, and training paradigm. A : Examples for the different tasks. Experiment tasks include mathematical reasoning and GUI agent task. B : Used LLMs include different series and parameter sizes. C : Our training process. For every task, questions are given to LLMs to generate a group of responses. The green arrow represents learning without rewards. The red arrow represents learning with rewards. Learning without rewards has long challenged traditional views of behavior. In a series of classic studies, Blodgett and Tolman demonstrated that rats exploring mazes without food rewards were nonetheless able to acquire knowledge of the maze layout ( 3 ; 36 ) . Once food was later introduced, their performance improved abruptly, revealing that substantial learning had taken place during the unrewarded exploration, a phenomenon that Tolman termed latent learning in psychology. These findings contradicted purely stimulus-response theories, which treat learning as reinforcement-driven association strengthening rather than the acquisition of internal representations of the environment. Consequently, the reward-centric learning often struggles to account for flexible and generalized behavior that emerges independently of direct reinforcement. Recent advances in reinforcement learning for LLMs have achieved impressive reasoning performance, yet these models remain fundamentally reward-dependent rather than exhibiting the latent learning. Advanced reasoning models such as OpenAI-o1 and DeepSeek-R1 have demonstrated remarkable capabilities across diverse tasks ( 2 ; 1 ; 11 ; 39 ; 35 ) . However, their learning frameworks remain deeply rooted in reward-centric paradigms, particularly reinforcement learning from outcome or process rewards and related optimization strategies ( 29 ; 31 ) . The prevailing assumption is that improvements in LLMs’ capabilities must be explicitly guided by reward signals. Yet, if biological agents can acquire predictive world models without direct reinforcement, this raises a fundamental question: can LLMs also exhibit latent learning, as in Tolman’s theory that learning occurs even without rewards, improving performance even without the guidance of reward signals and revealing further gains once rewards are introduced? Addressing this question not only establishes a conceptual bridge between cognitive psychology and LLMs but also provides a basis for understanding how LLMs may internalize knowledge without rewards. In this work, we provide novel empirical findings from our experiments that LLMs indeed display such latent learning as theorized by Tolman. We introduce an unrewarded exploration phase, analogous to Tolman’s latent learning, in which LLMs generate responses, omit response verification and rewards assignment, and subsequently learn directly from their own responses. Unrewarded exploration allows LLMs to explore and organize task-relevant knowledge without being constrained by reward-driven biases, leading to more coherent internal representations even in the absence of rewards, hence improving performance. Specifically, unrewarded exploration optimizes a KL-regularized policy matching objective that increases the likelihood of responses sampled from the current policy while constraining policy updates. Under this setting, we observe that LLMs trained without rewards already exhibit measurable improvements in performance. When rewards are subsequently introduced, these models demonstrate better performance gains compared to LLMs trained throughout with rewards. The resulting dynamics closely parallel Blodgett and Tolman’s experiments in rats ( 3 ; 36 ) , where learning happended during exploration without rewards and laid the groundwork for better performance improvements once food was provided. Beyond these surprising empirical findings, we further conduct theoretical analyses focusing on the unrewarded exploration phase in our experiments, explaining how models learn in the absence of rewards, thereby gaining performance improvement. To investigate whether latent learning arises in LLMs, we conduct experiments in two representative tasks for reinforcement learning for LLMs. First, we conduct experiments on mathematical reasoning task, a widely used task for reinforcement learning for LLMs, where performance improvements can be precisely measured through problem-solving accuracy ( 14 ; 8 ; 13 ; 17 ) . This domain provides a clean setting to examine whether performance can improve in the absence of reward signals and to quantify the extent of such improvement. Second, we conduct experiments on Graphic User Interface (GUI) agent tasks, where LLMs are required to act like human to manipulate computers, mobile devices, etc., extending our experiments to interactive environments ( 7 ; 37 ; 18 ; 19 ; 21 ; 6 ; 15 ) . These tasks capture two distinct levels of difficulty: in low-level tasks, the LLMs is required to perform a single correct action (i.e., to ground which area to perform click), while in high-level tasks, the model must execute a sequence of actions (e.g., click, type, close and etc.), adapting iteratively to changing environments until the goal is achieved. This distinction allows us to test whether the latent learning happends under both minimal and extended forms of sequential decision making, bridging simple reasoning with more complex agent behaviors. Together, these two domains provide complementary perspectives. Mathematical reasoning isolates unrewarded exploration in a controlled, symbolic setting, while GUI agent tasks demonstrate its emergence in embodied, interactive contexts. This combination ensures that our findings are not confined to a single task domain, but instead generalize across both static and dynamic reasoning environments. In summary, our contributions are as follows. • We introduce an unrewarded exploration phase for LLMs training that yields performance gains even in the absence of rewards, with further improvements once rewards are introduced. • We present theoretical analyses that provide mechanistic explanations for why LLMs can improve performance during unrewarded exploration. • We conduct extensive experiments across multiple model families, model parameter sizes, and diverse task domains to validate the existence of learning dynamics analogous to Tolman’s latent learning. 2 Related Work 2.1 Latent Learning in Psychology The phenomenon of latent learning was first discovered

Read the complete text