Exploitation Is All You Need… for Exploration
Micah Rentschler, Jesse Roberts · 2025
Read Exploitation Is All You Need… for Exploration by Micah Rentschler, Jesse Roberts (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01287.txt.
Introduction
Micah Rentschler and Jesse Roberts argue, in a 2025 paper, that pure greed can teach an agent to explore. Training agents that learn across many tasks usually needs built-in curiosity bonuses, but the authors claim exploration can emerge on its own under the right conditions.
Three ingredients are required: an environment with repeating structure, a memory the agent can consult, and credit that flows back over long horizons. In bandit problems and grid worlds, agents trained only to maximize reward began seeking information when structure and memory were present. Removing either ingredient killed the behavior in controlled ablations. The finding reframes exploration as a by-product, not an add-on.
Catalog record
| CREATOR | Micah Rentschler, Jesse Roberts |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploitation Is All You Need… for Exploration Micah Rentschler, Jesse Roberts Abstract Ensuring sufficient exploration is a central challenge when training meta-reinforcement learning (meta-RL) agents to solve novel environments. Conventional solutions to the exploration–exploitation dilemma inject explicit incentives such as randomization, uncertainty bonuses, or intrinsic rewards to encourage exploration. In this work, we hypothesize that an agent trained solely to maximize a greedy (exploitation-only) objective can nonetheless exhibit emergent exploratory behavior, provided three conditions are met: (1) Recurring Environmental Structure , where the environment features repeatable regularities that allow past experience to inform future choices; (2) Agent Memory , enabling the agent to retain and utilize historical interaction data; and (3) Long-Horizon Credit Assignment , where learning propagates returns over a time frame sufficient for the delayed benefits of exploration to inform current decisions. Through experiments in stochastic multi-armed bandits and temporally extended gridworlds, we observe that, when both structure and memory are present, a policy trained on a strictly greedy objective exhibits information-seeking exploratory behavior. We further demonstrate, through controlled ablations, that emergent exploration vanishes if either environmental structure or agent memory is absent (Conditions 1 & 2). Surprisingly, removing long-horizon credit assignment (Condition 3) does not always prevent emergent exploration—a result we attribute to the pseudo-Thompson Sampling effect . These findings suggest that, under the right prerequisites, exploration and exploitation need not be treated as orthogonal objectives but can emerge from a unified reward-maximization process. 1 Introduction Consider children playing hide-and-go-seek repeatedly. The seeker, eager to win, pays attention to where friends have hidden in past rounds. She remembers which spots were successful, which were not, and adapts her searching accordingly. She explores less familiar places not out of intrinsic curiosity, but because the knowledge she gains may help her tag more hiders and score more points in the long run. Can machines do the same? Can they learn to explore in order to exploit? That is the subject of this paper. We define exploration as the selection of actions with the aim of acquiring information about the environment’s dynamics or reward structure, thereby reducing epistemic uncertainty and enabling more effective decision-making. Exploration is ubiquitous in humans and animals, yet unlike drives such as hunger or pain, there is no clear, biologically plausible mechanism directly incentivizing it. Neuroscience shows the brain balances exploration and exploitation through meta-learning, which relies on repeating tasks, memory, and rewards ( Botvinick et al. 2018 ) . In reinforcement learning (RL), designers face a similar exploration–exploitation dilemma: how should an agent balance exploring unknown actions (potentially yielding future gain) and exploiting known rewards ( Sutton and Barto 2018 ; Thrun 1992 ) ? Traditional RL methods address this by introducing explicit exploratory incentives such as \epsilon -greedy sampling ( Sutton and Barto 2018 ) , Upper Confidence Bound (UCB) algorithms ( Auer, Cesa-Bianchi, and Fischer 2002 ) , or intrinsic motivation mechanisms like curiosity-driven exploration ( Pathak et al. 2019 ; Burda et al. 2018 ) that treat exploration as orthogonal to exploitation. In contrast, we ask: Can exploration emerge organically from a purely greedy objective? We hypothesize that it can—under three key conditions: 1. Recurring Environmental Structure. The task repeats so early-acquired information remains valuable later. 2. Agent Memory. The policy retains and utilizes past experiences across episodes. 3. Long-Horizon Credit Assignment. Learning must connect information gathering to long-term payoffs. We empirically test this hypothesis via controlled ablation studies in both multi-armed bandits and temporally extended gridworlds. With all three conditions met, our results show that the greedy agent consistently exhibits information-seeking behavior even without explicit incentives. Removing structure or memory eliminates this effect. Remarkably in some contexts, emergent exploration persists even when training optimizes only the immediate cumulative reward of the episode, suggesting a distinct alternate mechanism. Our findings challenge the view that exploration and exploitation require separate objectives. In structured environments with memory, reward maximization alone enables effective exploration. This suggests shifting focus from exploration bonuses to memory-rich architectures that leverage repeated patterns, simplifying RL design and offering a biologically plausible explanation for exploration. 2 Related Work The exploration—exploitation trade-off is a longstanding challenge in reinforcement learning (RL) and sequential decision making ( Sutton and Barto 2018 ; Lattimore and Szepesvári 2020 ; Thrun 1992 ) . Early approaches focused on randomization techniques, such as \epsilon -greedy or softmax action selection, as well as principled methods like Upper Confidence Bound (UCB) ( Auer, Cesa-Bianchi, and Fischer 2002 ) and Thompson Sampling (TS) ( Thompson 1933 ) , which provide regret guarantees. To extend tabular exploration to high-dimensional environments, density modeling and pseudo-count approaches were proposed ( Bellemare et al. 2016 ) . Hashing-based methods ( Tang et al. 2017 ) and successor representations ( Machado, Bellemare, and Bowling 2020 ) have shown strong performance in complex domains by encouraging exploration through intrinsic bonuses. Intrinsic motivation mechanisms, such as curiosity-driven exploration, reward agents for visiting novel or unpredictable states ( Oudeyer and Kaplan 2007 ; Pathak et al. 2019 ) . Random Network Distillation (RND) ( Burda et al. 2019 ) is a notable example, providing intrinsic bonuses in sparse-reward Atari games. Recent methods, such as SPIE ( Yu et al. 2023 ) , combine counts and trajectory structure to target bottlenecks in exploration. Meta-reinforcement learning (meta-RL) tackles rapid adaptation by learning across a distribution of related tasks ( Duan et al. 2017 ; Zintgraf et al. 2020 ) . Memory-rich architectures, such as recurrent neural networks ( Hausknecht and Stone 2015 ) and transformers ( Melo 2022 ; Chen et al. 2021 ; Parisotto et al. 2020 ) , allow policies to internalize exploratory strategies. In meta-RL, a recurrent policy successfully learns to explore via reward maximization alone ( Duan et al. 2017 ; Wang et al. 2017 ) . VariBAD ( Zintgraf et al. 2020 ) applies Bayesian principles to guide implicit exploration, though with explicit modeling of uncertainty. Neuroscientific evidence suggests that the brain’s meta-learning mechanisms manage the exploration–exploitation tradeoff by relying on specific prerequisites: recurring task structures, memory, and reward-driven learning processes ( Botvinick et al. 2018 ) . Notably, these prerequisites correspond closely to the conditions we have identified as essential for emergent exploration. In addition to presenting new empirical findings, this study clarifies the boundaries and limitations of emergent exploration in meta-reinforcement learning. We systematically delineate and validate the specific conditions necessary for exploration to emerge from pure exploitation objectives, emphasizing the critical roles of recurring environmental structure and agent memory. Our analysis further raises new questions about long-term credit assignment, suggesting that its necessity may be context-dependent. To our knowledge, this is the first work to clearly articulate and formalize the preconditions for emergent exploration, demonstrating that it arises in multi-step environments when these preconditions are met. This provides a concrete foundation for ideas that have long been discussed in the field. 3 Background 3.1 Repeated Markov Decision Processes (MDPs) To investigate when exploration emerges from pure exploitation, we consider a repeated-task setting: a fixed, partially observable Markov decision process (POMDP) instance M=(\mathcal{S},\mathcal{A},\mathcal{O},P,\Omega,r,\gamma) is sampled from a distribution, and the agent interacts with this same environment for multiple episodes. At the end of each episode, with probability 1/n , a new parameterization of the