AM-PPO: (Advantage) Alpha-Modulation with Proximal Policy Optimization

Soham Sane · 2025

Read AM-PPO: (Advantage) Alpha-Modulation with Proximal Policy Optimization by Soham Sane (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15514.txt.

Introduction

Soham Sane presents AM-PPO, an alpha-modulated form of proximal policy optimization, in a 2025 paper. PPO is a workhorse of reinforcement learning, prized for steady training. Its updates lean on advantage estimates, a score of how much better a move was than average. Those scores can swing wildly in size and noise, which slows or shakes the learning.

AM-PPO reshapes the scores as training goes. An alpha controller tracks the spread and size of the signals and sets a scaling factor to match. A tanh gate then bends the scores into a tame range. The reshaped signal, the tests show, gives smoother and faster learning than raw PPO.

Catalog record

CREATORSoham Sane
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15514

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AM-PPO: (Advantage) Alpha-Modulation with Proximal Policy Optimization Soham Sane Abstract Proximal Policy Optimization (PPO) is a widely used reinforcement learning algorithm that heavily relies on accurate advantage estimates for stable and efficient training. However, raw advantage signals can exhibit significant variance, noise, and scale-related issues, impeding optimal learning performance. To address this challenge, we introduce Advantage Modulation PPO (AM-PPO), a novel enhancement of PPO that adaptively modulates advantage estimates using a dynamic, non-linear scaling mechanism. This adaptive modulation employs an alpha controller that dynamically adjusts the scaling factor based on evolving statistical properties of the advantage signals, such as their norm, variance, and a predefined target saturation level. By incorporating a tanh \tanh -based gating function driven by these adaptively scaled advantages, AM-PPO reshapes the advantage signals to stabilize gradient updates and improve the conditioning of the policy gradient landscape. Crucially, this modulation also influences value function training by providing consistent and adaptively conditioned learning targets. Empirical evaluations across standard continuous control benchmarks demonstrate that AM-PPO achieves superior reward trajectories, exhibits sustained learning progression, and significantly reduces the clipping required by adaptive optimizers. These findings underscore the potential of advantage modulation as a broadly applicable technique for enhancing reinforcement learning optimization. 1 Introduction Policy gradient methods are a cornerstone of modern reinforcement learning (RL), with Proximal Policy Optimization (PPO) [ 1 ] being a particularly successful and widely adopted actor-critic algorithm. PPO learns a policy \pi_{\theta}(a|s) and a state-value function V_{\phi}(s) , leveraging advantage estimates A^{\pi}(s,a) to guide policy updates. While techniques like Generalized Advantage Estimation (GAE) [ 2 ] are instrumental in reducing the variance of these estimates, the resulting learning signal can still exhibit suboptimal scaling or residual noise, potentially hindering learning progress or leading to inefficiencies. Inspired by adaptive optimization methods such as Dynamic AlphaGrad (DynAG) [ 3 ] , which dynamically adjust learning signals based on their statistical properties, we propose AM-PPO (Alpha Modulation with PPO). AM-PPO enhances the standard PPO framework by introducing a novel mechanism to modulate raw advantage estimates prior to their use in the policy and value function learning updates. This modulation is governed by an adaptive controller whose internal state—most notably an adaptive scaling factor \alpha_{A} —evolves based on observed statistics of the advantages, such as their L_{2} norm, standard deviation, and a target saturation level for a non-linearly transformed version of these advantages. The central hypothesis of this work is that by dynamically rescaling and reshaping the advantage signal, AM-PPO can foster more stable and efficient policy learning. This is achieved by conditioning the policy gradient more effectively, potentially leading to improved optimization landscapes and more consistent learning trajectories. AM-PPO employs a specific modulation formula: it scales the absolute value of the raw advantages by a gating term. This gate is constructed from a shared scaling factor \kappa_{\text{shared}} and a \tanh function applied to the adaptively scaled, normalized advantages. Crucially, the state-value function V_{\phi}(s) in AM-PPO is also trained using targets derived from these modulated advantages, ensuring that both the actor and critic learn from a consistently transformed signal. This paper details the AM-PPO framework, elucidates its adaptive control mechanisms, discusses the theoretical and practical implications of its design, and presents an empirical evaluation of its performance characteristics. 2 PPO with Group Advantage Modulation (AM-PPO) AM-PPO retains the core structure of PPO but modifies how advantage estimates influence both the policy and value function updates. It employs a standard actor-critic setup and GAE for raw advantage calculation but introduces an adaptive modulation layer for these advantages, which are then used for both policy and value learning. 2.1 Preliminaries: PPO and Advantage Estimation The agent learns a policy \pi_{\theta}(a|s) and a state-value function V_{\phi}(s) . Data is collected by running the policy for N_{\text{steps}} in N_{\text{envs}} parallel environments. Raw advantage estimates A^{\text{raw}}_{t} are computed using GAE: A^{\text{raw}}_{t}=\sum_{l=0}^{N_{\text{steps}}-t-1}(\gamma\lambda)^{l}\delta_{t+l} (1) where \gamma is the discount factor, \lambda is the GAE smoothing parameter, and \delta_{t} is the TD error: \delta_{t}=r_{t}+\gamma V_{\phi}(s_{t+1})(1-d_{t})-V_{\phi}(s_{t}) (2) Here, r_{t} is the reward at timestep t , and d_{t} indicates if s_{t+1} is terminal. In standard PPO, the value function V_{\phi} is trained towards the on-policy target A^{\text{raw}}_{t}+V_{\phi}(s_{t}) . AM-PPO modifies this target as described later. 2.2 Adaptive Advantage Modulation Mechanism After calculating the raw GAE advantages \bm{A}_{\text{raw}} for a batch of experience, AM-PPO introduces an adaptive modulation step. This mechanism is governed by a controller with persistent Exponential Moving Average (EMA) states: \alpha_{A,\text{ema}} for an adaptive scaling factor, and s_{\text{prev},A,\text{ema}} for the observed saturation ratio of a transformed advantage signal. These EMA states are updated once per iteration based on the statistics of the entire batch of \bm{A}_{\text{raw}} . During the PPO update epochs, for each minibatch \bm{A}_{\text{raw},mb} , the modulation is applied using the frozen values of \alpha_{A,\text{ema}} and s_{\text{prev},A,\text{ema}} (i.e., their values at the start of the current iteration’s update phase). The transformation proceeds as follows: 1. Minibatch Statistics: Compute the L_{2} norm N_{A,mb}=\lVert\bm{A}_{\text{raw},mb}\rVert_{2} and standard deviation \sigma_{A,mb}=\Std(\bm{A}_{\text{raw},mb})+\epsilon_{A} . If N_{A,mb}<\epsilon_{A} , \bm{A}_{\text{mod},mb}=\bm{A}_{\text{raw},mb} and subsequent steps are skipped. 2. Read Frozen Adaptive Scaler: Let \alpha_{A,\text{current}}=\alpha_{A,\text{ema}} (the frozen value for this iteration’s updates). 3. Normalize and Scale for \tanh Input: Normalize the raw advantages by their L_{2} norm and scale by \alpha_{A,\text{current}} to get \bm{Z}_{A,mb} : \displaystyle\tilde{\bm{A}}_{\text{raw},mb} \displaystyle=\frac{\bm{A}_{\text{raw},mb}}{N_{A,mb}+\epsilon_{A}} (3) \displaystyle\bm{Z}_{A,mb} \displaystyle=\alpha_{A,\text{current}}\cdot\tilde{\bm{A}}_{\text{raw},mb} (4) 4. Apply Group Modulation Formula: The final modulated advantages \bm{A}_{\text{mod},mb} are computed using a shared scaling factor \kappa_{\text{shared}} ). The absolute value of the raw advantages is multiplied by a gating term: \bm{A}_{\text{mod},mb}=|\bm{A}_{\text{raw},mb}|\odot\left(\kappa_{\text{shared}}\cdot\tanh(\bm{Z}_{A,mb})\right) (5) The term M_{\text{gate}}=(\kappa_{\text{shared}}\cdot\tanh(\bm{Z}_{A,mb}) acts as a gate. The sign of the modulated advantage \bm{A}_{\text{mod},mb} is determined by the sign of this gate M_{\text{gate}} , while its magnitude is a product of |\bm{A}_{\text{raw},mb}| and |M_{\text{gate}}| . EMA State Updates (Once per Iteration): The persistent EMA states \alpha_{A,\text{ema}} and s_{\text{prev},A,\text{ema}} are updated once per training iteration, typically before the PPO update epochs begin, by applying the full controller logic to the entire batch of raw advantages \bm{A}_{\text{raw}} (or a large representative sample). This full logic includes: 1. Computing batch-wide N_{A} and \sigma_{A} . 2. Calculating a target alpha \hat{\alpha}_{A} using the controller’s shared scaling factor \kappa_{\text{shared}} (identical to the one in Eq. 5 ), the batch norm/std ratio, and saturation feedback: \hat{\alpha}_{A}=\kappa_{\text{shared}}\cdot\frac{N_{A}+\epsilon_{A}}{\sigma_{A}}\cdot\left(\frac{p_{\star,A}}{s_{\text{prev},A,\text{ema}}+\epsilon_{A}}\right)^{\eta_{A}} (6) 3. Updating \alpha_{A,\text{ema}} towards \hat{\alpha}_{A} via EMA with smoothing factor \rho_{A} , and clamping it: \alpha_{A,\text{ema}}\leftarrow\clip\big((1-\rho_{A})\alpha_{A,\text{ema}}+\rho_{A}\hat{\alpha}_{A},\alpha_{\min,A},\alpha_{\max,A}\big) (7) 4. Computing \bm{Z}_{A} for the batch using the newly updated \alpha_{A,\text{ema}} (as \alpha_{A,\text{current}} for this EMA update step). 5. Observing the current saturation for the batch s_{\text{curr},A}=\mathbb{E}\left[(\right]\lvert\bm{Z}_{A}\rvert>\tau_{A}) . 6. Updating s_{\text{prev},A,\text{ema}} towards s_{\text{curr},A} via EMA with smoothing factor \rho_{\text{sat},A} : s_{\text{prev},A,\text{ema}}\leftarrow(1-\rho_{\text{sat},A})s_{\text{prev},A,\text{ema}}+\rho_{\text{sat},A}s_{\text{curr},A} (8) 2.3 PPO Learning Update The PPO learning update proceeds for N_{\text{epochs}} over minibatches. (i) Normalize Modulated Advantages: If enabled by a configuration setting, the \bm{A}_{\text{mod},mb} from Equation 5 is further normalized to have mean zero and unit variance, yielding \hat{\bm{A}}_{\text{mod},mb} . Otherwise, \hat{\bm{A}}_{\text{mod},mb}=\bm{A}_{\text{mod},mb} . (ii) Policy Loss ( \mathcal{L}^{\pi} ): The policy parameters \theta are updated by maximizing the PPO clipped surrogate objective using \hat{\bm{A}}_{\text{mod},mb} . Let r_{j}(\theta)=\frac{\pi_{\theta}(a_{j}|s_{j})}{\pi_{\theta_{\text{old}}}(a_{j}|s_{j})} be the probability ratio. \mathcal{L}^{\pi}(\theta)=\mathbb{E}_{j\in mb}\left[\min\left(r_{j}(\theta)\hat{A}_{\text{mod},j},\clip(r_{j}(\theta),1-\epsilon_{\text{clip}},1+\epsilon_{\text{clip}})\hat{A}_{\text{mod},j}\right)\right] (9) (iii) Value Loss ( \mathcal{L}^{V} ): The value function parameters \phi are updated by minimizing the error between V_{\phi}(s_{j}) and the target V_{\text{target},j} . This target is constructed using the modulated advantages A_{\text{mod},j} (from Eq. 5 ,

Read the complete text