Refined Analysis of Entropy-Regularized Actor-Critic
Safwan Labbi, Paul Mangold, Daniil Tiapkin, Eric Moulines · 2026
Read Refined Analysis of Entropy-Regularized Actor-Critic by Safwan Labbi, Paul Mangold, Daniil Tiapkin, Eric Moulines (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24357.txt.
Introduction
Safwan Labbi and three colleagues examine the critic's role in actor-critic methods for entropy-regularized, finite, discounted settings. When the critic is exact, using it as a baseline cuts variance sharply, matching deterministic policy-gradient efficiency. With a learned critic, the actor update inherits the critic's variance and bias, though small critic error preserves the fast convergence. The practical moral is to learn the critic first and keep it current after each actor update.
Catalog record
| CREATOR | Safwan Labbi, Paul Mangold, Daniil Tiapkin, Eric Moulines |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 171161 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24357
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Refined Analysis of Entropy-Regularized Actor-Critic Safwan Labbi, Paul Mangold, Daniil Tiapkin, Eric Moulines Abstract In this paper, we study the role of the critic in actor–critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the latter as a baseline is a variance-reduction method in a strong sense. In this case, actor–critic with stochastic gradients matches the sample complexity of deterministic policy gradient, reaching an ϵ \epsilon -optimal regularized value with O ~ ( log ( 1 / ϵ ) ) \tilde{O}(\log(1/\epsilon)) samples. In practice, the critic is learned alongside the actor: the variance of the actor update is then influenced by the critic’s variance and bias. Specifically, when the critic has a sufficiently small error, the variance reduction and rapid convergence are preserved. This suggests to learn the critic first, keeping it up to date after each actor update, underscoring the crucial role of accurate critic estimation in actor–critic methods. 1 Introduction Policy gradient methods are among the most widely used reinforcement learning (RL) algorithms ( 45 ; 41 ) . Due to their inherent flexibility and scalability, they have emerged as the dominant approach in modern RL ( 1 ) . Their widespread adoption has led to the development of numerous techniques and tricks to stabilize training and accelerate convergence, enabling faster discovery of high-performing policies. A central technique in policy gradient methods is to introduce a baseline , which serves as a control variate that aims to reduce the variance of the gradient estimates ( 20 ) . This comes from the observation that, when expressing the gradient, any term that does not change the expected gradient direction can be used to stabilize optimization. In practice, there is a strong consensus on the fact that using the value function of the current policy strongly enhances performance ( 16 ; 39 ) . This effectively shifts the gradient’s focus from ’rewards’ to the so-called ’advantage’ of one action over another ( 2 ) , thereby providing a denser learning signal. This observation is further supported by the dominance of actor-critic methods (AC, 4 ; 20 ), and most specifically algorithms like A2C ( 28 ) , PPO ( 40 ) , TRPO ( 38 ) , among many others. Still, although this shift feels like a natural progression, the specific role of the value function as a stabilizer is poorly understood theoretically. Recently, the theory of RL has seen a surge of novel theoretical analyses, providing rigorous foundations for many fundamental methods. While earlier work relied on asymptotic guarantees ( 45 ; 15 ) , recent studies established non-asymptotic, global convergence rates for policy gradient methods ( 27 ; 46 ; 24 ) . More recently, theory has been extended to AC, proving global convergence in finite-time ( 22 ) , following analyses of policy gradient by relying on a uniform bound on the gradients. Parallel to these theoretical advances, it has recently been observed empirically that improving the critic considerably improves the convergence of AC ( 43 ) . While this gives strong evidence that AC reduces gradient variance, it is not clear to what extent. Specifically, we can distinguish between two types of variance reduction: • weak variance reduction : the variance of the updates is reduced by a multiplicative constant, akin to tail averaging ( 35 ) or moving average methods ( 30 ) ; • strong variance reduction : the variance of the gradient estimator vanishes as iterates approach the optimum, giving linear convergence, like SVRG ( 18 ) , SAGA ( 9 ) , and other methods. Naturally, strong variance-reduction results in much faster convergence rates than its weak counterpart. To our knowledge, it remains unknown whether AC methods achieve weak or strong variance reduction. Table 1 : Comparison with related actor-critic methods with unknown critic. Our method is the first to achieve \tilde{\mathcal{O}}(1/\epsilon) sample complexity. Type Algorithm Strong Variance Reduction Global Convergence Sample complexity Unregularized 34 ✗ ✗ \tilde{\mathcal{O}}(1/\epsilon^{2}) 22 ✗ ✓ \tilde{\mathcal{O}}(1/\epsilon^{3}) 13 ✗ ✓ \tilde{\mathcal{O}}(1/\epsilon^{3}) \text{Entropy-Regularized}^{(1)} 7 ✗ ✓ \tilde{\mathcal{O}}(1/\epsilon^{5}) Ent-AC (our work) ✓ ✓ \tilde{\mathcal{O}}(1/\epsilon) (1) The results here are provided for the entropy-regularized problem. In this paper, we answer positively: yes, AC achieves strong variance reduction , at least when the critic is known. Our theory follows from the observation that, when the value function (i.e., the critic) is perfectly known, the variance of AC’s stochastic gradient can be bounded by the norm of the deterministic gradient up to a multiplicative factor. In this case, we show that AC achieves \mathcal{O}(\log(1/\epsilon)) sample complexity, matching the deterministic iteration complexity. Alternatively, if the critic is learned online, we establish that AC’s behavior is governed by the critic’s bias and variance. Specifically, we show that if sufficiently many critic iterations are performed, AC attains \tilde{O}(1/\epsilon) sample complexity. Our contributions can be summarized as follows: • We propose a novel theoretical analysis of actor-critic. To the best of our knowledge, we provide the first proof that AC with an exact critic acts as a strong variance-reduction method , not merely reducing the residual variance of the updates, but entirely eliminating it. • When the critic is inexact, it must be learned alongside the actor. In that case, the critic’s bias and variance induce similar bias and variance on the actor. Consequently, when properly setting up the algorithm, one can learn the actor at a rate similar to the rate of learning the critic: in some sense, the most important part of the training is the critic, and it pays off to spend time learning it. Crucially, one needs to perform multiple updates of the critic in between actor updates, confirming recent empirical findings ( 43 ) . • We empirically confirm our findings on two environments, showing that the performance of the learned policy monotonically increases as the number of updates of the critic between consecutive actor updates increases. We give a comparison with the closest works in Table 1 , and discuss related work in Section 2 . Section 3 introduces the background. We analyze AC with exact critic in Section 4 , and with inexact critic in Section 5 . Empirical study is in Section 6 , and we discuss perspectives in Section 7 . 2 Related Works Entropy-Regularized RL. Entropy-regularized RL promotes stochastic policies by rewarding higher entropy, encouraging exploration, and improving stability in learning ( 44 ; 28 ; 33 ; 17 ) . It underpins widely used deep RL methods such as Soft Actor–Critic and entropy-regularized AC ( 17 ) , which remain poorly understood in theory despite their impressive practical performance. A line of work studies this objective both algorithmically and theoretically ( 31 ; 14 ) , relating entropy regularization to soft policy iteration and soft Q-learning, making explicit the links between policy-gradient and value-based methods. Another line of work ( 25 ) studies Policy Mirror Descent for entropy-regularized reinforcement learning and establishes convergence guarantees for the regularized objective. However, Policy Mirror Descent operates directly in policy space, which makes extensions beyond the tabular setting less straightforward. In contrast, actor–critic methods update parametrized policies directly, providing a more natural foundation for scalable extensions and motivating a dedicated analysis