ISOPO: Proximal policy gradients without pi-old

Nilin Abrahamsen · 2025

Read ISOPO: Proximal policy gradients without pi-old by Nilin Abrahamsen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23353.txt.

Introduction

Nilin Abrahamsen introduces ISOPO, a policy optimization method for reinforcement learning that approximates the natural policy gradient in a single step. Proximal methods like GRPO and CISPO approach the same target indirectly, taking multiple gradient steps while clipping importance ratios against a reference policy. ISOPO instead builds the geometry in directly, normalizing each sequence's log-probability gradient in the Fisher metric before combining it with the sequence's advantage.

A second variant reshapes the microbatch advantages using each layer's neural tangent kernel. Either way, the correction is applied layer by layer inside one backward pass, so the overhead over plain REINFORCE is negligible. The note is deliberately compact: its claim is that faithful natural-gradient geometry does not need the old-policy ratios, multiple epochs, and clipping machinery that current proximal methods carry. That simplicity is the selling point for practitioners who find ratio bookkeeping fragile.

Catalog record

CREATORNilin Abrahamsen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23353

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ISOPO: Proximal policy gradients without pi-old Nilin Abrahamsen Abstract This note introduces Isometric Policy Optimization (ISOPO), an efficient method to approximate the natural policy gradient in a single gradient step. In comparison, existing proximal policy methods such as GRPO or CISPO use multiple gradient steps with variants of importance ratio clipping to approximate a natural gradient step relative to a reference policy. In its simplest form, ISOPO normalizes the log-probability gradient of each sequence in the Fisher metric before contracting with the advantages. Another variant of ISOPO transforms the microbatch advantages based on the neural tangent kernel in each layer. ISOPO applies this transformation layer-wise in a single backward pass and can be implemented with negligible computational overhead compared to vanilla REINFORCE. 1 Introduction Reinforcement learning-based fine tuning has proven a powerful method to train reasoning and agentic behavior in LLMs. The GRPO algorithm and its variants ( [ 12 , 8 , 19 , 5 , 20 ] ) have become the de facto standard for RL-finetuning of LLMs and are an adaptation of the classical Proximal Policy Optimization (PPO) [ 11 ] algorithm with a simplified advantage estimator. Proximal policy methods such as GRPO can be viewed as efficient approximations to the natural policy gradient (NPG) [ 7 ] which itself would be impractical to implement as it involves the Fisher matrix whose size is quadratic in the number of parameters. Roughly, the NPG is defined to point towards the policy gradient but away from directions where the KL divergence grows the quickest. PPO/GRPO simulates an NPG step by taking multiple gradient steps with a surrogate objective based on a recent policy \pi_{\text{old}} such that large changes in log-probabilities are not incentivized. This effectively constrains the updates to remain in a trust region near \pi_{\text{old}} and stabilizes training. It is a simpler alternative to Trust-Region Policy Optimization (TRPO) [ 10 ] , which uses a different surrogate with a hard constraint on the estimated KL divergence. Figure 1 : ISOPO acts on the batch dimension of each layer before the advantage vector is applied to the log-probability gradients. Right: GRPO and related methods acts on the same batch dimension, but by clipping sequences based on a reference policy instead of based on the gradient. Bottom: Many optimizers act on the parameter dimension, including diagonal multiplers (AdamW) and non-diagonal transformations (K-FAC, Shampoo, SOAP, Muon). The natural policy gradient itself is not tractable. 1.1 Related work Several methods to improve RL-based LLM training focus on different re-weighting or clipping schemes for the sample-wise (tokenwise or sequence-wise) objective [ 17 , 18 , 3 ] ). CISPO [ 1 ] clips the sampling distribution to stabilize training without excluding tokens from the gradient calculation as in GRPO. Like these methods, ISOPO defines a re-weighting or transformation on the batch dimension (figure 1 right), but unlike the aforementioned methods, ISOPO computes the re-weighting based on gradient information. Because the transformation in ISOPO happens in each layer individually, the ISOPO gradient can be computed in a single backward without introducing additional computational cost. Natural gradient methods have been explored extensively in computational physics where the stochastic reconfiguration (SR) [ 14 ] method is analogous to the natural policy gradient, and the local energy is analogous to the advantage function. The MinSR [ 2 ] method improves efficiency of SR by preconditioning the batch dimension instead of the parameter dimension. Section 3 shows an interacting version of ISOPO which is analogous to a layer-wise version of the MinSR optimizer. Another class of approximate natural gradient methods act on the parameter gradient vectors, typically layer-wise (figure 1 bottom). These include the K-FAC [ 9 ] , Shampoo [ 4 ] , and SOAP [ 16 ] . These methods pre-condition the rows and columns of a linear layer’s weight matrix separately, an approximation that gives rise to the Kronecker-factored (K-FAC) name. Another related optimizer is Muon [ 6 , 15 ] which unlike K-FAC operates only on the reduced gradients. Since these optimizers act on the parameter dimension, they can in principle be combined with ISOPO similarly to how AdamW and Muon optimizers are used with clipping-based GRPO. The experiments in this note use AdamW for easy comparison with the VeRL baselines [ 13 ] . 1.2 Refresher on the natural policy gradient Recall that the Fisher matrix F is defined such that for a perturbation v of the parameter vector \theta , the squared Fisher metric \|v\|_{F}=v\cdot Fv is the squared perturbation of the log-probability, averaged over samples. That is, v\cdot Fv=\mathbb{E}[|\nabla_{\theta}\log\pi_{\theta}(o|q)\cdot v|^{2}], (1) where q is the question and o is the output of the model. This is to leading order the KL divergence incurred by the perturbation (or \chi^{2} divergence, which is easier to see). The natural policy gradient is v=F^{-1}g where g=\mathbb{E}[A(q,o)\nabla_{\theta}\log\pi_{\theta}(o|q)] is the vanilla policy gradient. Here A(q,o) is the advantage of output o . Importantly, the transformation F^{-1} that defines the natural policy gradient does not involve the advantage A(q,o) . That is, F^{-1} penalizes parameter directions with large policy moves, but it does not penalize large advantages. 2 The non-interacting ISOPO gradient update The idea of the non-interacting ISOPO is to estimate the size of the gradient of each output sequence in the Fisher metric, i.e. how much each sequence contributes to the policy movement in terms of KL divergence. The sequences are then normalized in the Fisher metric before being multiplied by their advantages. This transformation is applied layer-wise during the backward pass (algorithm 2 ). ⬇ 1 # Compute non-interacting ISOPO microbatch gradient for a linear layer. 2 # Run backprop on objective = torch.sum(log_prob) without advantages. 3 def compute_ISOPO_grad_full_bwd_hook ( mod , _ , batch_grad_out ): 4 5 # Get samples to estimate Fisher norm sqrt(v * F v) 6 overlap_samples = torch . randperm ( microbatch_grad_out . shape [0])[: n_overlap_samples ] 7 a0 = act_in [ overlap_samples ] 8 g0 = g_out [ overlap_samples ] 9 10 # Reduce and re-scale sequence-wise 11 grad = 0.0 12 for seq_id , seq_indices in enumerate ( microbatch . indices_by_sequence ): 13 act_in = mod . microbatch_act_in [ seq_indices ] # activations registered in forward pass 14 grad_out = microbatch_grad_out [ seq_indices ] 15 seq_logprob_grad = grad_out . T @ act_in 16 17 F_norm = ( 18 torch . norm ( torch . sum (( g0 @ seq_logprob_grad ) * a0 , dim =1)) / 19 torch . norm ( torch . norm ( g0 , dim =1) * torch . norm ( a0 , dim =1)) # optionally cache this 20 ) 21 grad += microbatch . advantages [ seq_id ] * rescaling ( seq_logprob_grad , F_norm = F_norm ) 22 23 # accumulate 24 mod . grad += grad 25 26 def rescaling ( grad , F_norm ): 27 # use e.g. EMA of trailing minibatch means for the regularization 28 return 1 / ( F_norm **2 + regularization ). sqrt () Figure 2 : The non-interacting variant of ISOPO scales the sequence gradients based on a stochastic estimate of the Fisher norm of the log-probability gradient. 2.0.1 Derivation: Sequence-wise bound on the KL contribution The re-scaling is motivated as follows: Let v=\nabla_{\theta}\log\pi_{\theta}(o|q) be the

Read the complete text