Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
Yixian Zhang, Huaze Tang, Changxu Wei, Wenbo Ding · 2025
Read Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning by Yixian Zhang, Huaze Tang, Changxu Wei, Wenbo Ding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01639.txt.
Introduction
Yixian Zhang and co-authors wrote a 2025 paper on a two way version of a leading reinforcement learning method. The classic method updates its policy by chasing a target distribution in one direction of a standard gap measure, which forces messy approximations. For bell shaped policies, the other direction has a clean closed form.
Their bidirectional version starts from that clean projection, then refines with the classic direction. On standard control benchmarks it beat the classic and other rivals, with top scores up to about a third higher and better use of samples. The trick costs little: a smarter first step.
Catalog record
| CREATOR | Yixian Zhang, Huaze Tang, Changxu Wei, Wenbo Ding |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01639
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning Yixian Zhang, Huaze Tang, Changxu Wei, Wenbo Ding Abstract The Soft Actor-Critic (SAC) algorithm, a state-of-the-art method in maximum entropy reinforcement learning, traditionally relies on minimizing reverse Kullback-Leibler (KL) divergence for policy updates. However, this approach leads to an intractable optimal projection policy, necessitating gradient-based approximations that can suffer from instability and poor sample efficiency. This paper investigates the alternative use of forward KL divergence within SAC. We demonstrate that for Gaussian policies, forward KL divergence yields an explicit optimal projection policy—corresponding to the mean and variance of the target Boltzmann distribution’s action marginals. Building on the distinct advantages of both KL directions, we propose Bidirectional SAC, an algorithm that first initializes the policy using the explicit forward KL projection and then refines it by optimizing the reverse KL divergence. Comprehensive experiments on continuous control benchmarks show that Bidirectional SAC significantly outperforms standard SAC and other baselines, achieving up to a 30 % 30\% increase in episodic rewards, alongside enhanced sample efficiency. 1 Introduction Maximum entropy reinforcement learning has emerged as a powerful framework for tackling complex decision-making problems, including mobile robot control ( de Jesus et al. 2021 ) , autonomous driving ( Yang et al. 2024 ) , and protein engineering ( Wang et al. 2023 ) . The Soft Actor-Critic (SAC) algorithm ( Haarnoja et al. 2018a ) , a prominent algorithm within this framework, has demonstrated superior performance across a variety of continuous benchmark tasks. In SAC, the optimization of the policy is achieved by seeking a projection policy with respect to the Boltzmann distribution based on the Q-function, where the reverse Kullback-Leibler (KL) divergence is used as the probabilistic measure. However, due to the intractability of the projection policy, gradient-based methods are employed to iteratively adjust the parameterized policy towards the optimal projection ( Haarnoja et al. 2018b ) . Despite their utility, these optimization techniques can exhibit instability and require a substantial number of samples to achieve convergence ( Chan et al. 2022 ; Banerjee et al. 2022 ) . In fact, even in the simple case of a Gaussian distribution, we observe that gradient optimization methods often fail to converge. The use of reverse KL divergence as a measurement to guide policy projection has two key advantages, as outlined in prior studies ( Haarnoja et al. 2018b ; Duan et al. 2021b ; Hu et al. 2025 ) : First, minimizing the reverse KL divergence ensures that the expected Q-function value of the updated policy is greater than that of the previous policy. Second, the gradient of the reverse KL divergence is computationally tractable, avoiding the need for the intractable normalization term in the Boltzmann distribution. However, if the reverse KL divergence optimization becomes unstable, such as when the KL divergence increases during gradient optimization, these advantages are lost. An alternative approach involves the forward KL divergence, which is commonly utilized in variational policy optimization ( Dayan and Hinton 1997 ; Toussaint and Storkey 2006 ) , where few papers concentrate on its use in the SAC algorithm. In this paper, we find that, by replacing the reverse KL divergence with the forward KL divergence in the SAC framework, denoted as Forward SAC, the optimal projection policy becomes explicit. Specifically, for Gaussian policies—widely used in stochastic continuous actor-critic algorithms ( Haarnoja et al. 2018a ; Schulman et al. 2017 ; Mnih et al. 2016 ; Schulman et al. 2015 ) —the optimal projection policy within forward KL divergence corresponds to the mean and variance of the action under the Boltzmann distribution. This insight allows us to directly compute the optimal projection policy using numerical integration, thereby achieving stable and sample-efficient policy updating. However, it is not guaranteed that the updated policy within forward KL divergence can maximize the expected Q value. These characteristics of the two directions of KL divergence lead to a direct idea: Can we integrate the advantages of these two directions of KL divergence? Therefore, we propose the Bidirectional SAC algorithm, which uses the optimal projection policy in forward KL divergence as the initialization policy, and it optimizes the policy by minimizing the reverse KL divergence. To efficiently compute the mean and variance, we adapt the Value Decomposition Networks (VDN) ( Sunehag et al. 2017 ) , originally designed for multi-agent reinforcement learning, into the single-agent context for Q-function learning, where we denote it as VDN-a networks. Our theoretical analysis, grounded in Copula theory, elucidates how the VDN-a architecture is designed to approximate the necessary marginal properties of the Boltzmann distribution. Supported by experimental results, this approach facilitates the efficient learning required for the independent numerical integration of the mean and variance for each action dimension. Comparative experiments in MuJoCo and Box2D environments ( Todorov et al. 2012 ; Catto 2011 ) demonstrate that the Bidirectional SAC algorithm outperforms the SAC algorithm and other baseline methods, achieving up to 30\% increase in episodic rewards. Our contributions are as follows: • We introduce the Forward SAC algorithm, in which the updated policy is the optimal projection to the Boltzmann distribution of the Q-function. • We conduct a thorough analysis for the use of forward and reverse KL divergence in the SAC algorithm, where we find that the use of both directions has its advantages and disadvantages. To leverage these advantages, we propose the Bidirectional SAC algorithm, which optimizes the policy sample efficiently. • To efficiently compute the optimal projection, we propose VDN-a networks for single-agent Q-function learning. Our theoretical analysis and experimental results demonstrate its efficacy in approximating the marginals of the Boltzmann distribution, crucial for efficiently computing the forward KL optimal projection within Bidirectional SAC. Our experiments demonstrate that the Bidirectional SAC algorithm converges rapidly and outperforms baseline algorithms, achieving up to 30\% increase in episodic rewards. 2 Preliminaries and background In this section, we briefly introduce the maximum entropy reinforcement learning and the famous soft actor-critic (SAC) algorithm ( Haarnoja et al. 2018a ) . 2.1 Maximum entropy reinforcement learning We consider policy search in an infinite-horizon Markov decision process \langle\mathcal{S},\mathcal{A},p,r,\rho\rangle , with continuous state \mathcal{S} and action \mathcal{A} space. The transition function p:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,\infty) defines the probability density of transitioning, and rewards are r_{t}=r(s_{t},a_{t})\in[r_{\min},r_{\max}] . The objective of reinforcement learning is to learn an optimal policy \pi^{*} that maximizes the cumulative expected rewards \pi^{*}=\arg\max_{\pi}\mathbb{E}_{\tau\sim\pi}\left[\sum_{l=t}r_{l}\right] , where \tau denotes the trajectory under policy \pi originating at (s_{t},a_{t}) ( Sutton and Barto 2018 ) . The maximum entropy reinforcement learning extends the standard objective by incorporating an entropy term \mathcal{H}(\pi(\cdot\mid s))=-\mathbb{E}_{a\sim\pi(\cdot\mid s)}\left[\log\pi(a\mid s)\right] , thereby encouraging policies to maintain stochasticity and explore more effectively. In this setting, the optimal policy \pi_{\text{Ent}}^{*} seeks to maximize both the expected cumulative rewards and the entropy of the policy at each state ( Haarnoja et al. 2018b ) : \pi_{\text{Ent }}^{*}=\arg\max_{\pi}\mathbb{E}_{\tau\sim\pi}\left[\sum_{l=t}r_{l}+\alpha\mathcal{H}\left(\pi\left(\cdot\mid s_{l}\right)\right)\right], (1) where \alpha is the temperature parameter to determine the relative importance of entropy and reward. 2.2 Soft actor-critic Equation ( 1 ) is addressed by various methods ( Ziebart et al. 2008 ; Abdolmaleki et al. 2018a ; Kim et al. 2024 ) . Soft Q-Learning (SQL) ( Haarnoja et al. 2017 ) , an early Q-learning approach, used