PROMA: Projected Microbatch Accumulation for Reference-Free Proximal Policy Updates
Nilin Abrahamsen · 2026
Read PROMA: Projected Microbatch Accumulation for Reference-Free Proximal Policy Updates by Nilin Abrahamsen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10498.txt.
Introduction
Nilin Abrahamsen gives a new train rule for agents that learn from rewards. The rule is named PROMA. Past top rules grade each act next to a fixed ref model. That ref can bind the agent and cost a full model in memory. PROMA drops the ref.
It works at the level of small batches. As slopes add up across batches, it trims the sum. The trim cuts the part that lines up with the batch at hand. The trim is done layer by layer in the back pass. That keeps it cheap to run. The aim is firm updates with no ref model.
Catalog record
| CREATOR | Nilin Abrahamsen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 10 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10498
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PROMA: Projected Microbatch Accumulation for Reference-Free Proximal Policy Updates Nilin Abrahamsen Abstract This note introduces Projected Microbatch Accumulation (PROMA), a proximal policy method that modifies gradient accumulation across microbatches rather than relying on likelihood ratios relative to a reference policy. During accumulation, PROMA projects the partially accumulated gradient to be orthogonal to the sequence-wise gradients of the current microbatch. This projection is applied layer-wise during the backward pass, enabling efficient implementation. A within-microbatch variant (Intra-PROMA) acts independently across microbatches. Empirically, PROMA achieves proximal updates without entropy collapse while providing tighter local KL control than GRPO. 1 Background Proximal Policy Optimization (PPO) [ 9 ] and its variants, including GRPO [ 10 ] , stabilize reinforcement learning by limiting the magnitude of policy updates between optimization steps. This is typically achieved by optimizing a surrogate objective that discourages large deviations from a reference policy \pi_{\text{old}} through clipping of the policy likelihood ratio. GRPO and related methods [ 10 , 14 ] combine PPO-style updates with a group-relative advantage estimator and have become the dominant approach for reinforcement learning–based fine-tuning of large language models. CISPO [ 3 ] is another proximal policy variant that clips the importance-sampling ratio in the loss while ensuring that all tokens contribute to the gradient computation. More broadly, proximal policy methods can be viewed as tractable approximations to the natural policy gradient [ 7 ] , whose exact computation is infeasible at scale. 2 Projected Microbatch Accumulation (PROMA) PROMA modifies the optimization process during the backward pass. For each microbatch, the sequence-wise log-probability gradients ( mcb_seq_grads in Algorithm 2 ) are computed as part of evaluating the policy gradient, denoted mcb_policy_grad . Before gradients are accumulated, the running gradient t.grad of a parameter tensor t is updated by projecting out components that lie in the span of the sequence-wise log-probability gradients. This projection is applied layer-wise and acts directly on the accumulated gradient. ⬇ 1 def accumulate_PROMA ( t , mcb_policy_grad , mcb_seq_grads ) 2 "" " 3 k: number of sequences in microbatch 4 t: (d,) tensor of model weights in the layer 5 mcb_policy_grad: (d,) vanilla policy gradient w.r.t. tensor t for the microbatch 6 mcb_seq_grads: (d, k), sequence-wise gradients of log_prob.sum() w.r.t tensor t 7 " "" 8 complemented_acc_grad = project_to_complement ( t . grad , mcb_seq_grads ) 9 t . grad = complemented_acc_grad + mcb_policy_grad 10 11 def project_to_complement ( acc_vec , vecs ): 12 Q , _ = torch . linalg . qr ( vecs , mode = "reduced" ) # Q: (d, k) 13 return acc_vec - Q @ ( Q . T @ acc_vec ) Figure 1 : accumulate_PROMA is called during the backward pass of the policy gradient loss and modifies the gradient accumulation. Before adding the microbatch gradient, the partially accumulated gradient is projected to be orthogonal to the gradient of each sequence in the current microbatch ( mcb_seq_grads ). The motivation for PROMA can be put simply: \nabla\log\pi(s) for a sequence s represents the parameter direction where the sequence’s probability changes the fastest. The parameter update d\theta represented by t.grad in Algorithm 2 incurs a KL divergence that grows with the squared overlaps \mathbb{E}_{s}[(d\theta\;\cdot\;\nabla_{\theta}\log\pi(s))^{2}] . In Algorithm 2 , each column of mcb_seq_grads represents such a sample \nabla_{\theta}\log\pi(s))^{2} . PROMA aims to avoid this contribution to the KL divergence by projecting away these components of the gradient update. Efficiency To exactly project away the k sequences in the microbatch takes \approx 2k^{2}d FLOPS where d is the number of model parameters. If the microbatch size is large then this can be applied group-wise so that k\sim 8 . The cost 2k^{2}d of PROMA is then negligible compared to the T\times d operations required to form the gradient, where T is the number of tokens. Another option is to apply an approximate projection as in Algorithm 4 which takes O(kd) operations. 3 An intra-microbatch variant (Intra-PROMA) While PROMA modifies gradient accumulation across microbatches, the Intra-PROMA method defined below instead acts within each microbatch. Because the projection depends only on the current microbatch, Intra-PROMA is embarrassingly parallel across microbatches and compatible with standard data-parallel training. ⬇ 1 def proma_intra ( token_advantages , act_in , grad_out , r =100, shrinkage =1.0): 2 "" " 3 Project out gradient components in the subspace spanned by 4 top-r singular vectors of activations and gradient outputs. 5 " "" 6 r = min ( r , act_in . shape [0], act_in . shape [1], grad_out . shape [1]) 7 8 Q_a = approx_rank_r_basis ( act_in , r ) # (d_in, r) 9 Q_g = approx_rank_r_basis ( grad_out , r ) # (d_out, r) 10 11 policy_grad = ( token_advantages * grad_out . T ) @ act_in / len ( token_advantages ) 12 projection = Q_g @ ( Q_g . T @ policy_grad @ Q_a ) @ Q_a . T 13 return policy_grad - shrinkage * projection 14 15 def approx_rank_r_basis ( X , r , power_iters =1): 16 "" "Approximate top-r right singular vectors via randomized SVD." "" 17 n = X . shape [0] 18 omega = torch . randn ( n , r ) 19 Y = X . T @ omega 20 for _ in range ( power_iters ): 21 Y = X . T @ ( X @ Y ) 22 Q , _ = torch . linalg . qr ( Y ) 23 return Q Figure 2 : accumulate_PROMA is called during the backward pass of the policy gradient loss and modifies the gradient accumulation. Before adding the microbatch gradient, the partially accumulated gradient is projected to be orthogonal to the gradient of each sequence in the current microbatch ( mcb_seq_grads ). To avoid the cost of operating on full gradient matrices, Intra-PROMA separately approximates the dominant subspaces of activations and gradient outputs. This is similar to the Kronecker factorizations which underlie K-FAC [ 8 ] and related second-order optimizers [ 12 ] , which apply a sandwich product G^{-1}(\cdot)A^{-1} to precondition the gradient. In contrast, PROMA uses the factored subspaces for projection rather than preconditioning. I have found that applying the sandwich projection to the subtracted component: \texttt{grad}-Q_{g}Q_{g}^{\top}\texttt{grad}\,Q_{a}Q_{a}^{\top} (1) controls KL without impeding training progress. Alternatively one could apply a sandwich product of projections to the gradient ( 2 ) similarly to the G^{-1}(\cdot)A^{-1} in K-FAC: (I-Q_{g}Q_{g}^{\top})\,\texttt{grad}\,(I-Q_{a}Q_{a}^{\top}),\qquad\neq\text{PROMA} (2) but I found that ( 2 ) caused the validation score to plateau at a lower score. 4 Results The demonstration was implemented in VeRL [ 11 ] ( https://github.com/nilin/proma ). Experiments used the GSM8K dataset [ 5 ] with the Qwen-3 0.6B model [ 13 ] . The PROMA configuration is based on REINFORCE without any PPO clipping of the likelihood ratio, to isolate the effect of the projected accumulation. All runs are without KL penalty. GRPO refers to the baseline with PPO clipping, and REINFORCE refers to the baseline without PPO clipping. All runs use the same group-relative advantage estimator. Figure 3 shows the performance and policy dynamics of PROMA compared to GRPO and REINFORCE. (a) Validation performance (b) KL vs. initial policy (c) Entropy (d) KL vs. lagged policy Figure 3 : Training performance and policy statistics for PROMA, GRPO, and REINFORCE. (a) Validation performance. (b)