GIPO: Gaussian Importance Sampling Policy Optimization

Chengxuan Lu, Zhenquan Zhang, Shukuan Wang, Qunzhi Lin, Baigui Sun, Yang Liu · 2026

Read GIPO: Gaussian Importance Sampling Policy Optimization by Chengxuan Lu, Zhenquan Zhang, Shukuan Wang, Qunzhi Lin, Baigui Sun, Yang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03955.txt.

Introduction

Chengxuan Lu and others propose GIPO, a policy tuning objective improving data speed for post-training multimodal agents with reinforcement learning. GIPO starts from truncated importance sampling, swapping hard clipping for a Gaussian trust weight based on log ratios that gently damps extreme ratios while keeping nonzero gradients. Theoretical analysis shows an implicit tunable constraint on update magnitude, with concentration bounds guaranteeing robustness and balance under finite-sample gauging. Tests show leading results among clipping rivals, with strong balance and sample speed.

Catalog record

CREATORChengxuan Lu, Zhenquan Zhang, Shukuan Wang, Qunzhi Lin, Baigui Sun, Yang Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58826
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GIPO: Gaussian Importance Sampling Policy Optimization Chengxuan Lu, Zhenquan Zhang, Shukuan Wang, Qunzhi Lin, Baigui Sun, Yang Liu Abstract Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation. However, RL remains limited by poor data efficiency, particularly in settings where interaction data are scarce and quickly become outdated. To address this challenge, GIPO (Gaussian Importance sampling Policy Optimization) is proposed as a policy optimization objective based on truncated importance sampling, replacing hard clipping with a log-ratio-based Gaussian trust weight to softly damp extreme importance ratios while maintaining non-zero gradients. Theoretical analysis shows that GIPO introduces an implicit, tunable constraint on the update magnitude, while concentration bounds guarantee robustness and stability under finite-sample estimation. Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias–variance trade-off, high training stability and improved sample efficiency. 1 Introduction In many real-world reinforcement learning (RL) applications, such as robotic control, healthcare decision-making, and industrial automation, interacting with the environment is prohibitively expensive or time-consuming. Consequently, strict on-policy learning, which requires fresh data for every parameter update, is often impractical due to limited throughput and data scarcity. To improve sample efficiency, training pipelines inevitably adopt asynchronous execution or heavy reliance on experience replay, reusing historical trajectories to maximize the utility of every interaction. However, this necessity introduces a fundamental challenge: the experience stored in the replay buffer is generated by older behavior policies that lag behind the current learner policy ( 2 ; 18 ) . This policy lag creates a distribution mismatch, causing importance ratios between the target and behavior policies to exhibit heavy-tailed distributions as variability accumulates over time. Two broad approaches are commonly used to handle this regime. One line adopts fully off-policy control (e.g. DQN/SAC-style actor–critic methods), which is explicitly designed to learn from replayed experience ( 15 ; 5 ) . Another line retains PPO-family conservative policy optimization as the backbone, extending it to replay-heavy settings to trade strict on-policy updates for higher throughput and practical stability ( 2 ; 11 ) . Given our target setting’s large stochastic policies, sparse rewards, and high interaction costs, we follow the second route. Fully off-policy methods can be sensitive to critic learning instability under distribution shift, often requiring extensive tuning ( 3 ; 8 ) . By contrast, PPO-family objectives provide a mature, conservative surrogate with robust stability monitors (e.g., trust regions) and a direct actor update, making them a preferred backbone for reliable post-training. However, standard stabilizers for policy lag do not necessarily translate into the effective reuse of stale replay. While target-side corrections (like V-trace ( 2 ) ) improve value estimation, they do not address a distinct actor-side bottleneck. When importance ratios become heavy-tailed due to lag, standard PPO-style surrogates employ a “hard clipping” mechanism to constrain updates. In replay-heavy training, this mechanism often becomes overly aggressive, effectively zeroing out the gradient contribution of many stale samples. This leads to a phenomenon of utilization collapse: valuable but stale trajectories are replayed and processed computationally, yet they contribute almost nothing to the policy update, resulting in significant data inefficiency in cost-sensitive domains. We therefore propose GIPO ( G aussian I mportance sampling P olicy O ptimization) to use stale replay more effectively. GIPO replaces the hard thresholding of standard methods with a smooth trust weighting in log-ratio space. Specifically, it applies a Gaussian kernel to the log-importance ratios, providing a symmetric, differentiable damping of extreme samples. This approach yields controlled down-weighting of off-policy data while preserving non-zero gradients in the tails. By doing so, GIPO allows stale replay to contribute small but informative updates rather than being effectively discarded, turning historical data into a useful signal without sacrificing stability. Our contributions are as follows: • We propose GIPO, a smooth log-ratio trust-weighted surrogate for PPO-style policy optimization. It is specifically designed to mitigate utilization collapse under policy lag, enabling more effective learning from stale replay data. • Our theoretical analysis demonstrates that GIPO implicitly enforces a tunable bound on the policy update magnitude. We theoretically prove and empirically validate that GIPO achieves a superior bias-variance trade-off compared to baseline methods, providing formal guarantees of robustness and stability under finite-sample estimation. • We evaluate GIPO on Meta-World ( 25 ) tasks and the LIBERO ( 10 ) benchmark using the 7B OpenVLA-OFT backbone ( 7 ) in two regimes (fresh vs. stale). This large-scale study consumes over 10,000 H200 GPU-hours, and results show that GIPO achieves superior sample efficiency and effective replay usage compared to clipping-based baselines, particularly when data freshness is limited. 2 Related Work In scalable policy learning, experience is often generated by a behavior policy \mu that is stale relative to the learner policy \pi_{\theta} , which is referred to as policy lag. Such mismatch is widespread in distributed actor-learner systems and can also occur in nominally on-policy methods due to reusing rollouts across multiple optimization epochs ( 18 ; 2 ) . Under lag, updates become effectively off-policy and are mediated by per-step importance ratios. In high-dimensional stochastic policies and long-horizon settings, these ratios can become broadly distributed and exhibit pronounced tails. These tails are closely tied to both optimization instability and reduced effective contribution of stale replay under conservative surrogates. Existing work primarily employs two approaches to address this: (i) target-side off-policy correction via truncated importance sampling, and (ii) surrogate-side ratio control. A broader related line studies how to reuse trajectories across policy iterations via importance sampling with explicit variance/bound control, including policy-search and natural-policy-gradient settings ( 13 ; 14 ; 19 ; 9 ) . 2.1 Truncated Off-policy Correction under Policy Lag A classical response to behavior-target mismatch is importance sampling (IS), but naive IS can incur prohibitive variance when \mu and \pi_{\theta} diverge. A widely adopted principle is therefore truncated importance sampling, which trades controlled bias for substantially reduced variance. Retrace( \lambda ) is a canonical example for multi-step bootstrapping under off-policy drift: it uses per-decision truncated ratios in the return recursion, with a \lambda -style trace that geometrically down-weights distant steps, yielding a stable bias–variance trade-off for multi-step value targets ( 16 ) . In distributed actor-learner training, IMPALA introduces V-trace, which applies truncated per-decision ratios when constructing multi-step value targets and the corresponding policy-gradient advantages, explicitly motivated by actor-learner delay ( 2 ) . Concretely, V-trace uses two clipped coefficients: one for policy-gradient correction and one for trace propagation, so that both the immediate correction and the multi-step credit assignment remain bounded. Replay-heavy actor-critic methods such as ACER further integrate truncation with bias correction and conservative updates to safely reuse stale trajectories ( 24 ) . While these methods substantially improve the reliability of value and advantage estimation under policy lag, they primarily act on the target construction side. Under heavy-tailed ratios, conservative policy surrogates can still yield weak actor updates on stale replay. Even with stabilized advantages, tail-ratio samples may be strongly down-weighted or enter saturated regions under ratio-based objectives. Consequently, the effective update may become dominated by a small subset of transitions due to contribution degeneracy. This motivates complementary modifications at the surrogate level.

Read the complete text