Generative Actor-Critic with Soft Bridge Policies

Shunpu Tang, Yafei Wang, Lisheng Fan · 2026

Read Generative Actor-Critic with Soft Bridge Policies by Shunpu Tang, Yafei Wang, Lisheng Fan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08733.txt.

Introduction

Ke He, Le He, and three co-authors build SoftGAC, a reinforcement learning method whose actor generates expressive actions in a single network pass. Diffusion and flow actors capture complex action patterns but hide their densities and demand costly repeated evaluations with unstable long backpropagation chains. SoftGAC instead uses one stochastic feedforward network with internal Gaussian bridge layers, making the soft objective exact in path space.

An analytic regularizer compares the actor against a high entropy reference process for both improvement and value learning, with no endpoint entropy estimate needed. At parameter budgets matching strong rivals, returns on hard continuous control tasks match or beat diffusion and flow policies at far lower inference cost.

Catalog record

CREATORShunpu Tang, Yafei Wang, Lisheng Fan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08733

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generative Actor-Critic with Soft Bridge Policies Shunpu Tang, Yafei Wang, Lisheng Fan Abstract Expressive generative policies such as diffusion and flow models are appealing for maximum-entropy (MaxEnt) online reinforcement learning because of their ability to model multimodal and highly non-Gaussian action distributions. However, training effective soft generative policies faces two obstacles that often arise together. First, marginal action densities are often unavailable, so existing methods typically rely on entropy bounds, heuristic proxies or approximations. Second, iterative shared-parameter samplers raise inference cost and require backpropagation through time over repeated network evaluations, increasing memory cost and destabilizing policy optimization. These obstacles motivate us to seek a generative policy that exposes a tractable MaxEnt objective while requiring only a single sampled actor forward pass for action generation. To this end, we propose soft generative actor-critic ( SoftGAC ), whose actor defines a stochastic bridge from a fixed base latent to a terminal action latent in pre-tanh space. This structured bridge allows us to lift the MaxEnt objective as an analytically tractable path-wise relative-entropy objective against a high-entropy reference process. In practical finite-step implementation, this relative entropy reduces exactly to sampled transition control energy and thus provides principled soft regularization. Moreover, we keep the single-pass actor lightweight by using small step-specific bridge transitions, each evaluated only once per sampled action, while maintaining a parameter budget comparable to strong actor baselines. Extensive experiments on challenging continuous-control benchmarks show that SoftGAC attains higher or competitive returns than strong generative policy baselines, including diffusion and flow-matching policies, while staying in the low-latency regime of one-pass actors and avoiding the much higher cost of many-step diffusion samplers, showing considerable improvements in the compute-return tradeoff. Code and resources https://github.com/skypitcher/soft_gac_jax 1 Introduction Maximum-entropy (MaxEnt) online reinforcement learning (RL) has become a central approach for off-policy continuous-control because it combines value improvement with explicit stochasticity [ 11 , 19 , 9 , 8 ] . In soft actor-critic (SAC) [ 8 ] , this stochasticity is not only an exploration device. It is part of the optimization objective, where the actor is encouraged to choose high-value actions while maintaining high entropy [ 9 ] . This principle is simple and effective when the policy has a tractable density, but the usual Gaussian policy class can be too restrictive for complex control problems that may require multimodal or highly non-Gaussian action distributions [ 17 , 5 , 12 , 20 ] . Expressive generative policies offer a natural way to expand the actor class [ 10 , 25 , 13 , 16 , 23 ] . Diffusion, score-based and flow-style policies can represent rich action distributions and have recently become attractive for reinforcement learning [ 21 , 6 , 29 , 14 , 30 ] . Their appeal, however, creates a tension with MaxEnt RL. The soft actor update is an endpoint-density objective that requires policy entropy or an equivalent density-based regularizer. For many generative actors, the marginal action density is unavailable or expensive to evaluate because the action is obtained after marginalizing over latent variables or generation paths. Existing methods therefore often introduce entropy bounds [ 4 ] , entropy estimators and approximations [ 28 , 6 ] , noise-augmented path likelihoods [ 31 ] or Wasserstein-geometric proxies [ 15 , 32 ] , which make the connection to MaxEnt RL indirect. A second difficulty comes from how many generative policies spend computation. Diffusion or flow-style samplers with many denoising or refinement steps usually reuse the same network across time [ 23 , 4 , 21 , 6 ] . This raises inference cost at deployment and it also makes the number of function evaluations (NFE) a training-time cost, because the actor update differentiates Q(s,a) through the sampled action and every refinement step that produced it [ 18 ] . Increasing NFE therefore increases BPTT depth, activation memory and actor-update wall-clock [ 7 ] . It can also lengthen gradient paths through repeated uses of shared parameters, which may destabilize actor optimization. As a result, high-NFE policies may achieve stronger performance, but the added optimization burden often limits their marginal gains [ 6 ] . Conversely, reducing NFE lowers both training and inference cost, but can cause a sharp performance drop [ 20 , 4 ] . These two obstacles motivate us to seek a different design principle. Specifically, we seek a generative actor that retains the expressiveness of stochastic latent generation while exposing a tractable soft objective and avoiding a long shared-parameter sampler. The goal is not to replace repeated sampler evaluations with a larger network that hides the cost in parameters. Instead, we ask whether careful actor structure can match or even exceed the performance of high-NFE diffusion policies with a single sampled forward pass and a parameter budget comparable to strong actor baselines. Our key idea is to make the actor a lightweight, explicit path law over latent variables, rather than treating it only through its terminal action distribution. This path-law view addresses the soft-regularization problem by replacing endpoint entropy with path-wise relative entropy to a high-entropy reference process. Its bridge structure addresses the computation problem by using a small number of lightweight step-specific transitions that are evaluated once per action under a compact actor parameter budget, rather than repeatedly applying a shared sampler. The resulting path Kullback-Leibler (KL) divergence refines the endpoint MaxEnt principle because it contains an endpoint KL term to the reference terminal action law. It also adds a principled regularizer on how the actor reaches the terminal action. We instantiate this idea as soft generative actor-critic ( SoftGAC ), an off-policy actor-critic method with soft bridge policies, as demonstrated in Figure 1 . The parameter-efficient actor uses a short sequence of lightweight local Gaussian transitions to form a stochastic bridge in pre-tanh latent space from a fixed base latent to a terminal action latent. These explicit local transitions make the path-wise relative entropy to the reference bridge analytically tractable. For the finite-step bridge used in the algorithm, its trainable part reduces exactly to a sampled transition-control-energy objective. The resulting actor update directly trades off critic value against this sampled control energy, while action generation requires one sampled pass through the bridge blocks. Our contributions are three-fold. (i) We formulate a path-space soft objective for generative actors, show that it contains the endpoint KL regularizer used by MaxEnt RL as a marginal component and characterize both its unrestricted endpoint-equivalent optimum and the effect of using a practical fixed actor base. (ii) We design a soft bridge policy that uses a short sequence of local Gaussian transitions in pre-tanh latent space, making the path-wise regularizer exactly computable as finite-step control energy and enabling a direct actor update with action generation by a single sampled bridge pass under a comparable actor parameter budget. (iii) We demonstrate on challenging continuous-control benchmarks that SoftGAC improves the compute-return tradeoff by pairing strong returns with low one-pass action-generation cost and a parameter budget comparable to strong actor baselines. Figure 1: Overview of the proposed soft bridge policy. Local KL terms compare actor and reference transitions and sum to the sampled control energy. The terminal latent is mapped through \tanh and optimized by the critic. Appendix C provides a 2D bridge visualization.

Read the complete text