CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning
Yexin Li, Pring Wong, Hanfang Zhang, Shuo Chen, Siyuan Qi · 2025
Read CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning by Yexin Li, Pring Wong, Hanfang Zhang, Shuo Chen, Siyuan Qi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18980.txt.
Introduction
Yexin Li and co-authors published this paper in 2025. It addresses exploration, the stubborn problem of reinforcement learning: an agent that only exploits what it knows never discovers better moves. Their method, CAE, turns the critic, the network that already judges actions, into the explorer.
Borrowing tricks from the multi-armed bandit literature, CAE nudges the agent toward uncertain, promising actions, scaled in a way that carries proof of sublinear regret. The change needs about ten lines of code and adds no new parameters. For tasks where the judge itself is hard to train, an extension adds a tiny helper network, under one percent more parameters. Tests on robot control, game, and navigation benchmarks showed stable, strong exploration. The authors pitch the method as a rare mix of theory, simplicity, and results that practitioners can adopt quickly.
Catalog record
| CREATOR | Yexin Li, Pring Wong, Hanfang Zhang, Shuo Chen, Siyuan Qi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18980
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning Yexin Li, Pring Wong, Hanfang Zhang, Shuo Chen, Siyuan Qi Abstract Exploration remains a critical challenge in reinforcement learning, as many existing methods either lack theoretical guarantees or fall short of practical effectiveness. In this paper, we introduce CAE, a lightweight algorithm that repurposes the value networks in standard deep RL algorithms to drive exploration without introducing additional parameters. CAE utilizes any linear multi-armed bandit technique and incorporates an appropriate scaling strategy, enabling efficient exploration with provable sub-linear regret bounds and practical stability. Notably, it is simple to implement, requiring only around 10 10 lines of code. In complex tasks where learning an effective value network proves challenging, we propose CAE + + , an extension of CAE that incorporates an auxiliary network. This extension increases the parameter count by less than 1 % 1\% while maintaining implementation simplicity, adding only about 10 additional lines of code. Experiments on MuJoCo and MiniHack show that both CAE and CAE + + outperform state-of-the-art baselines, bridging the gap between theoretical rigor and practical efficiency. 1 Introduction Exploration in reinforcement learning (RL) remains a fundamental challenge, particularly in environments with complex dynamics or sparse rewards. Although algorithms such as DQN ( Mnih et al. 2015 ) , PPO ( Schulman et al. 2017 ) , SAC ( Haarnoja et al. 2018 ) , DDPG ( Lillicrap et al. 2016 ) , TD3 ( Fujimoto et al. 2018 ) , IMPALA ( Espeholt et al. 2018 ) , and DSAC ( Duan et al. 2021 ; Duan et al. 2023 ) have demonstrated impressive performance on tasks like Atari games ( Mnih et al. 2013 ; Mnih et al. 2015 ) , StarCraft ( Vinyals et al. 2019 ) , Go ( Silver et al. 2017 ) , etc. , they often depend on rudimentary exploration strategies. Common approaches, such as \epsilon -greedy or injecting noise into actions, are typically inefficient and struggle in scenarios with delayed or sparse rewards. For decades, exploration with proven optimality in tabular settings has been available ( Kearns & Singh 2002 ) . More recently, methods with provable regret bounds have been developed for scenarios involving function approximation, including linear functions ( Osband et al. 2016 ; Osband et al. 2019 ; Jin et al. 2018 ; Jin et al. 2020 ; Agarwal et al. 2020 ) , kernels ( Yang et al. 2020 ) , and neural networks ( Yang et al. 2020 ) . However, while linear and kernel-based approaches make strong assumptions about the RL functions, provable methods based on neural networks suffer from prohibitive computational costs, specifically O(n^{3}) , where n is the number of parameters in the RL network. Moreover, some studies ( Ash et al. 2022 ; Ishfaq et al. 2024a ) propose algorithms with theoretical guarantees under the linearity assumption and attempt to extend them directly to deep RL settings without further proof. Other works ( Ishfaq et al. 2021 ; Ishfaq et al. 2024b ) provide provable bounds for deep RL, but they are either practically burdensome or rely on an unknown sampling error. A more practical approach to exploration relies on heuristics, leading to the development of several empirically successful methods, including Pseudocount ( Bellemare et al. 2016 ) , ICM ( Pathak et al. 2017 ) , RND ( Burda et al. 2019b ) , RIDE ( Raileanu & Rocktäschel 2020 ) , NovelD ( Zhang et al. 2021a ) , AGAC ( Flet-Berliac et al. 2021 ) , and E3B ( Henaff et al. 2022 ; Henaff et al. 2023 ) . These methods typically use internally generated bonuses to encourage agents to explore novel states based on specific metrics. For example, RND ( Burda et al. 2019b ) leverages the prediction error of a randomly initialized target network as the exploration bonus, while RIDE ( Raileanu & Rocktäschel 2020 ) combines errors from forward and inverse dynamics models to compute the bonus. However, these methods introduce bias to the external rewards from the environment, lack theoretical guarantees, and are primarily guided by intuitive heuristics. Additionally, they often require training extra networks beyond the standard value or policy networks used in deep RL algorithms, resulting in increased computational overhead. Figure 1: Comparison of existing exploration methods, such as E3B ( Henaff et al. 2022 ) , with CAE and CAE + . Here, L_{b} represents the Bellman loss used to update the state- or action-value function, while L_{f} refers to the loss of the auxiliary network. E3B requires additional networks to generate exploration bonuses, while CAE utilizes the embedding layers of the value network for bonuses, resulting in reduced computational overhead and no additional parameters. CAE + extends CAE by incorporating a small auxiliary network to enhance performance in sparse reward environments, with only a minor increase in parameters. In this work, we aim to combine the strengths of both theoretically grounded and empirically effective exploration methods. Provably efficient exploration methods are fundamentally rooted in the theory of Multi-Armed Bandits (MAB) ( Li et al. 2010 ; Chu et al. 2011 ; Agrawal & Goyal 2013 ; Wen et al. 2015 ; Zhang et al. 2021b ; Zhou et al. 2020 ) . Building on this foundation, we hypothesize that advanced techniques from neural MAB can be effectively adapted for exploration in deep RL . Existing studies ( Zahavy & Mannor 2019 ; Riquelme et al. 2018 ; Xu et al. 2022 ) indicate that decoupling deep representation learning from exploration strategies shows promise for achieving efficient exploration in neural MAB. Motivated by these insights, we propose CAE. Unlike existing methods that train additional embedding networks to generate exploration bonuses, CAE leverages the embedding layers of the value network in the RL algorithm and employs linear MAB techniques to produce exploration bonuses. To ensure the practical stability and functionality of CAE, we adopt an appropriate scaling strategy ( Welford 1962 ; Elsayed et al. 2024 ) to process the bonuses, as elaborated in Section 3.2 . Consequently, CAE introduces no additional parameters beyond those in the original algorithm, showcasing that RL algorithms inherently possess strong exploration capabilities if their learned networks are effectively leveraged. Moreover, CAE is simple to implement, requiring only about 10 lines of code. A comparison between CAE and existing exploration methods is in Figure 1 . For tasks with complex dynamics or sparse rewards, learning an effective value network is challenging, often impeding exploration based on it. Accordingly, we propose an extended version CAE + , as illustrated in Figure 1 . CAE + integrates a small auxiliary network to facilitate the learning process. The structure of the auxiliary network is carefully designed to prevent severe coupling between the environment dynamics and returns, thereby further enhancing the practical performance of CAE + . Remarkably, this addition results in less than a 1\% increase in parameter count and requires only about 10 additional lines of code. We evaluated CAE and CAE + on the MuJoCo and MiniHack benchmarks to assess their effectiveness in both dense and sparse reward environments. CAE consistently improves the performance of state-of-the-art baselines such as PPO ( Schulman et