Actor-Critic without Actor
Donghyeon Ki, Hee-Jun Ahn, Kyungyoon Kim, Byung-Jun Lee · 2025
Read Actor-Critic without Actor by Donghyeon Ki, Hee-Jun Ahn, Kyungyoon Kim, Byung-Jun Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21022.txt.
Introduction
Actor-critic methods, a workhorse of reinforcement learning, need two networks that are fussy to tune together. Donghyeon Ki and three co-authors remove one of them in a 2025 paper called Actor-Critic without Actor. Their framework, ACA, drops the explicit actor network entirely.
Separate actor and critic networks make training sensitive to architecture choices and hyperparameters, which hurts scaling. Diffusion models have been tried as expressive actors, but they add cost and design burden. ACA instead derives the policy improvement step without a dedicated actor, keeping the critic's evaluation role. The authors present the framework as lightweight and easier to deploy. The work speaks to practitioners who find actor-critic tuning, rather than the task itself, to be the bottleneck.
Catalog record
| CREATOR | Donghyeon Ki, Hee-Jun Ahn, Kyungyoon Kim, Byung-Jun Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21022
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Actor-Critic without Actor Donghyeon Ki, Hee-Jun Ahn, Kyungyoon Kim, Byung-Jun Lee Abstract Actor-critic methods constitute a central paradigm in reinforcement learning (RL), coupling policy evaluation with policy improvement. While effective across many domains, these methods rely on separate actor and critic networks, which makes training vulnerable to architectural decisions and hyperparameter tuning. Such complexity limits their scalability in settings that require large function approximators. Recently, diffusion models have recently been proposed as expressive policies that capture multi-modal behaviors and improve exploration, but they introduce additional design choices and computational burdens, hindering efficient deployment. We introduce Actor-Critic without Actor (ACA), a lightweight framework that eliminates the explicit actor network and instead generates actions directly from the gradient field of a noise-level critic. This design removes the algorithmic and computational overhead of actor training while keeping policy improvement tightly aligned with the critic’s latest value estimates. Moreover, ACA retains the ability to capture diverse, multi-modal behaviors without relying on diffusion-based actors, combining simplicity with expressiveness. Through extensive experiments on standard online RL benchmarks, ACA achieves more favorable learning curves and competitive performance compared to both standard actor-critic and state-of-the-art diffusion-based methods, providing a simple yet powerful solution for online RL. 1 Introduction Actor-critic methods represent a foundational paradigm in reinforcement learning (RL), in which a critic estimates action values under the current policy and an actor updates the policy toward higher-value actions ( Sutton et al. 1998 ; Konda & Tsitsiklis 1999 ) . This alternating cycle of evaluation and improvement is theoretically grounded and has demonstrated strong empirical success across diverse domains ( Mnih et al. 2016 ; Lowe et al. 2017 ; Haarnoja et al. 2018a ; Haarnoja et al. 2018b ; Espeholt et al. 2018 ) . However, the alternating updates increase algorithmic complexity, requiring careful tuning of network architectures and learning rates for stability ( Andrychowicz et al. 2021 ) , while doubling computation and memory demands, making actor-critic methods less attractive in domains requiring large function approximators ( Ouyang et al. 2022 ; Rafailov et al. 2024 ) . Moreover, the gradual policy updates required by the actor contrast with Q -learning’s direct maximization of the critic, resulting in slower policy improvement as the actor cannot instantly incorporate the critic’s latest estimates. Recent advances have introduced diffusion models as powerful policy parameterizations for RL ( Wang et al. 2022 ; Chen et al. 2022 ; Lu et al. 2023 ; Chen et al. 2023 ; Chen et al. 2024 ; Zhu et al. 2024 ; Zhang et al. 2024 ; Ren et al. 2024 ; Lu et al. 2025 ) . These models generate actions by progressively denoising Gaussian noise over a sequence of timesteps, enabling expressive multi-modal action distributions well-suited for complex control. In the offline setting, this expressivity enables diffusion policies to recover high-return trajectories from heterogeneous datasets and surpass unimodal Gaussian policies. This benefit extends to online RL as well, where diffusion policies promote broader exploration and better mode coverage ( Wang et al. 2024 ; Yang et al. 2023 ; Psenka et al. 2023 ; Ding et al. 2024 ) . Although diffusion-based policies provide strong expressivity for modeling complex, multi-modal action distributions, their deployment in online RL introduces substantial practical challenges. In particular, they rely on large denoising networks that significantly increase memory consumption and training time, and often require additional approximations that introduce bias into policy updates ( Ma et al. 2025 ) . These factors complicate implementation, exacerbate computational overhead, and ultimately limit scalability in settings where efficient and lightweight adaptation is crucial. To address these limitations, we introduce Actor-Critic without Actor (ACA) , a lightweight framework that eliminates the explicit actor network and relies solely on the critic. Inspired by guidance techniques in diffusion-based offline RL, ACA reformulates action sampling as a critic-guided denoising process, where actions are obtained directly from the gradient field of a noise-level critic. A key distinction of ACA is that it preserves the multi-modal behavior inherent to diffusion models without requiring a separately parameterized, computationally heavy actor. Removing the actor not only reduces model complexity but also ensures that diverse action modes are faithfully represented through the critic alone. Moreover, this mechanism replaces standard policy improvement with gradient-based refinement, keeping sampled actions remain aligned with up-to-date value estimates and thereby eliminating the policy lag. Through this design, ACA achieves improved learning curves and competitive performance on MuJoCo continuous control benchmarks compared to both standard actor-critic methods and diffusion-based approaches, while requiring substantially fewer parameters. Figure 1: Comparison between standard actor-critic methods and the proposed Actor-Critic without Actor (ACA). Standard methods maintain both an actor and a critic, adding complexity and overhead, whereas ACA eliminates the actor and achieves policy improvement via critic-guided denoising. 2 Preliminaries Reinforcement learning (RL) We consider the RL problem under a Markov Decision Process (MDP) \mathcal{M}=\{\mathcal{S},\mathcal{A},P,r,\gamma,p_{0}\} , where \mathcal{S} is the state space, \mathcal{A} is the action space, P(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a}) is the transition probability, r(\mathbf{s},\mathbf{a}) is the reward, p_{0} is the initial-state distribution, and \gamma\in[0,1) is the discount factor. The goal of RL is to learn a policy that maximizes the expected cumulative discounted reward in MDP. For a policy \pi(\mathbf{a}|\mathbf{s}) , the state-action value is defined as Q^{\pi}(\mathbf{s},\mathbf{a})=\mathbb{E}\left[\sum_{\tau=0}^{\infty}\gamma^{\tau}r(\mathbf{s}_{\tau},\mathbf{a}_{\tau})|\mathbf{s}_{0}=\mathbf{s},\mathbf{a}_{0}=\mathbf{a},\pi,P\right] , where \tau denotes environment timesteps in RL. A central principle of RL is the policy iteration framework, which alternates between two steps. First, policy evaluation estimates Q^{\pi} for a fixed policy, typically by iterating the Bellman operator \displaystyle(\mathcal{T}^{\pi}Q)(\mathbf{s},\mathbf{a})=r(\mathbf{s},\mathbf{a})+\gamma\,\mathbb{E}_{\mathbf{s}^{\prime}\sim P(\cdot|\mathbf{s},\mathbf{a}),\mathbf{a}^{\prime}\sim\pi(\cdot|\mathbf{s}^{\prime})}\left[Q(\mathbf{s}^{\prime},\mathbf{a}^{\prime})\right]. (1) Second, policy improvement updates \pi toward actions that maximize the expected Q -value, e.g., \pi(\cdot|\mathbf{s})\leftarrow\arg\max_{\mathbf{a}}Q(\mathbf{s},\mathbf{a}) . Actor-critic methods instantiate this paradigm in a parametric form: the critic approximates Q^{\pi} via Bellman backups, while the actor is updated using the critic’s value estimates, thereby coupling policy evaluation and improvement in a single learning loop. Denoising diffusion probabilistic models (DDPMs) DDPMs ( Ho et al. 2020 ) are a class of generative models that construct samples through a Markov forward-reverse process. In the forward process, Gaussian noise is incrementally added to a clean data sample \mathbf{x}_{0} over T timesteps, according to q(\mathbf{x}_{t}|\mathbf{x}_{t-1}):=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}) , where \beta_{t}\in(0,1) is a predefined variance schedule and t denotes the diffusion timestep. Importantly, this process admits a closed-form expression for sampling \mathbf{x}_{t} from \mathbf{x}_{0} at any timestep t , q(\mathbf{x}_{t}|\mathbf{x}_{0}):=\mathcal{N}(\mathbf{x}_{t};\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0},(1-\bar{\alpha}_{t})\mathbf{I}), with \alpha_{t}:=1-\beta_{t} and \bar{\alpha}_{t}:=\prod^{t}_{s=1}\alpha_{s} . The reverse process starts from standard Gaussian noise \mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) and progressively denoises through a parameterized Markov chain: \displaystyle\mathbf{x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(\mathbf{x}_{t},t)\right)+\sigma_{t}\mathbf{z},\quad\text{where}\quad\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}), (2) where the variance is fixed as \sigma_{t}^{2}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t} . The diffusion model \epsilon_{\theta} is trained to approximate the added noise by minimizing a simplified surrogate objective derived from a variational bound: \displaystyle\mathbb{E}_{\mathbf{x}_{0}\sim\mathcal{B},\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}),t\sim\mathcal{U}[1,T]}\left[\left\|\epsilon-\epsilon_{\theta}\left(\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,t\right)\right\|^{2}\right]. (3) Classifier guidance Diffusion models incorporate guidance mechanisms that bias the generative process toward samples aligned with a desired label ( Ho & Salimans 2022 ; Dhariwal & Nichol 2021 ) . Classifier guidance ( Dhariwal & Nichol 2021 ) trains a noise-level classifier p_{\phi}(y|\mathbf{x}_{t},t) to predict labels from noisy inputs \mathbf{x}_{t} , and its gradient \nabla_{\mathbf{x}_{t}}\log p_{\phi}(y|\mathbf{x}_{t},t) is used to steer the diffusion sampling process toward the target class y . The guided noise prediction is defined as \hat{\epsilon}(\mathbf{x}_{t},t):=\epsilon_{\theta}(\mathbf{x}_{t},t)-w\sigma_{t}\nabla_{\mathbf{x}_{t}}\log p_{\phi}(y|\mathbf{x}_{t},t) , where w>0 denotes a guidance weight. Here, \epsilon_{\theta} is the noise-prediction network of the diffusion model, and \hat{\epsilon} is the guided variant