Generative OOD-regularized Model-based Policy Optimization

Aysin Tumay 1 Jiahe Huang 1 Elise Jortberg 2 Rose Yu 1 1 University of California, San Diego 2 Abiomed · 2026

Read Generative OOD-regularized Model-based Policy Optimization by Aysin Tumay 1 Jiahe Huang 1 Elise Jortberg 2 Rose Yu 1 1 University of California, San Diego 2 Abiomed (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24405.txt.

Introduction

Tumay and three co-authors tackle offline reinforcement learning, where sparse-data policies may pick out-of-distribution actions. To keep policies safe, they use density estimation where data is thin. Their algorithm, GORMPO, restricts policy updates to high-density areas. They compare density estimators on detection and performance, using a medical dataset and offline datasets. Empirically, it beats baselines by 17 percent on the medical dataset and improves the base model on offline datasets. Better detection generally yields better policies where dynamics are stable, while conservative penalties win otherwise.

Catalog record

CREATORAysin Tumay 1 Jiahe Huang 1 Elise Jortberg 2 Rose Yu 1 1 University of California, San Diego 2 Abiomed
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101484
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24405

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generative OOD-regularized Model-based Policy Optimization Aysin Tumay 1 Jiahe Huang 1 Elise Jortberg 2 Rose Yu 1 1 University of California, San Diego 2 Abiomed Abstract We study sequential decision-making with offline reinforcement learning (RL). Traditional offline RL policies may result in out-of-distribution (OOD) actions when training relies only on sparse offline representations. To ensure safe offline policies in a sparse state-action space, we explore how density estimation models can be integrated into model-based RL methods to avoid the OOD regions. Generative models are capable of explicitly modeling the density in sparse state-action spaces. Building on this, we introduce G enerative O OD- r egularized M odel-based P olicy O ptimization (GORMPO), a density-regularized offline RL algorithm that uses generative density modeling to restrict policy updates to high-density areas of the dataset. Furthermore, we examine whether better OOD detection corresponds to better model-based offline policies . We compare (1) the OOD detection capabilities of various density estimators and (2) their performance within the GORMPO framework on a real-world medical dataset and sparse offline RL datasets. We theoretically guarantee GORMPO’s performance under mild assumptions. Empirically, GORMPO outperforms state-of-the-art baselines by 17% on a real-world medical dataset and enhances the base model on the offline RL datasets. Our empirical findings show that better OOD detection generally results in improved policies in environments with stable dynamics, while conservative penalties with poor density estimation are favored when dynamics are uncertain. 1 Introduction Offline reinforcement learning (RL) has shown great promise in safety-critical control tasks where interaction with an online environment is impossible. Applications of this problem have been demonstrated in sepsis 22 ; 32 and cancer prediction 12 ; 38 , autonomous driving 3 , and Unmanned Aerial Vehicle control 5 . However, distribution shift constitutes a significant problem for offline RL because the policy reaches out-of-distribution (OOD) regions, leading to unsafe actions. Offline RL also depends strongly on dataset coverage where limited state–action support leads to extrapolation error in Bellman backups 23 . This issue is common in clinical settings because most patients receive adequate support and remain hemodynamically stable, while cases of insufficient treatment are rare and underrepresented as seen in Figure 1 . Figure 1 : Left: reward–action space of our medical dataset with the sparse region in gray. Right: the transition rollout distribution versus train distribution modeled by KDE and NeuralODE. While transition rollouts exhibit low likelihoods indicating OOD behavior, KDE saturates in OOD regions with a large spike. A core challenge in offline RL is mitigating distribution shift without being overly conservative. Prior model-free methods constrain the learned policy to the behavior distribution on the offline data only 14 ; 24 . However, these constraints potentially suppress valid actions where the Q-function generalizes well 2 . Other model-free methods 42 ; 28 address this by explicitly regularizing the policy with behavior support based on the evidence lower bound (ELBO), or with importance sampling 28 , which might fall short on highly sparse offline data. Some model-based uncertainty-penalized methods 46 ; 37 , built on top of model-based policy optimization (MBPO) 18 , address distribution shift by penalizing policy optimization with uncertainty. However, augmenting the dataset with model-generated rollouts can quickly drift into OOD states when the dynamics model is trained on sparse data and fails to generalize reliably. Other works explored MBPO with kernel density estimation (KDE) regularization in healthcare 45 ; 39 . However, generative density estimation offers a more principled alternative for flexible OOD detection compared to KDE, especially in sparse and high-stakes tasks such as healthcare 47 ; 29 ; 35 . In this work, we propose G enerative O OD- r egularized M odel-based P olicy o ptimization ( GORMPO ), a density-penalized model-based offline RL algorithm. To avoid the uncovered areas in the state-action space during dynamics model rollouts, we discount the rolled-out rewards based on the next state-action density. Lower-density areas get penalized more to support in-distribution (ID) data augmentation during policy optimization. On top of this, since our pipeline is highly flexible, we calibrated five different density estimation models to answer if better OOD detection lead to better offline policies when used in GORMPO. In summary, our primary contributions are: • We propose GORMPO, a plug-and-play framework for any model-based offline RL algorithm that explicitly penalizes OOD state-action rollouts with generative models. GORMPO outperforms state-of-the-art baselines by 17% on our proprietary real-world medical dataset. • We provide the first comprehensive evaluation of 5 distinct families of density estimators with four generative models for the OOD detection task. • We demonstrate that expressive density models lead to better policies in stable dynamics, whereas pessimistic regularization is required for uncertain dynamics. 2 Related Work Constrained Offline RL. Addressing distribution shift is central to offline reinforcement learning. A prominent class of methods mitigates extrapolation error by constraining the learned policy to the support of the behavior policy. Approaches such as Batch-Constrained Q-learning (BCQ) 14 , Conservative Q-Learning (CQL) 24 , and Behavior Regularized Actor-Critic (BRAC) 43 employ divergence metrics to penalize deviations from the offline dataset. While effective at ensuring safety, distance-based constraints often induce excessive conservatism. By restricting learning to the neighborhood of observed data, they may suppress potentially optimal actions that remain plausible within the true underlying distribution 2 ; 9 . To overcome these limitations, recent work has shifted to explicit regularization in offline RL. Model-free methods regularize the policy toward the behavior distribution 42 , reinforcing the patterns in the offline dataset during optimization. However, estimating the lower bound provides a coarse regularization. With explicit density estimation in model-free RL, CPED 48 integrates FlowGAN, highlighting the benefits of generative density models. Nevertheless, relying on the data support with weak expert demonstrations can still yield suboptimal policies. In this case, MBPO 18 proves that learning a dynamics model helps in broader exploration of the state-action space. Some impose pessimism through uncertainty penalties 37 ; 46 , quantifying uncertainty across an ensemble of dynamics models and penalizing value estimation and dynamics rollouts. LEQ 30 further addresses value overestimation in model rollouts via lower expectile regression of \lambda -returns. SAMBO-RL 27 introduces a shift-aware reward correction to mitigate model bias. However, uncertainty penalties, bias corrections, and value regularization can still be insufficient if rollouts drift into OOD regions when the dynamics model is trained on sparse data, requiring a density constraint. Density regularization in model-based RL with kernel density estimators such as OGSRL 45 and CORMPO 39 has demonstrated success in suppressing OOD policies. However, the kernel density estimator is not expressive in high-dimensional settings. For MBPO, density regularization with an expressive generative model for transition rollouts is essential in sparse state-action domains. Generative Density Estimators. Generative density estimation has advanced through three primary methodological paradigms. First, normalizing flows 33 construct complex distributions from simple base measures via sequences of invertible transformations, facilitating exact likelihood evaluation. Second, diffusion and score-based generative models 36 model distributions through a forward noise-injection and learned reverse process. Recent iterations, such as improved DDPMs 16 and EDM 20 , have further optimized log-likelihood estimation and sample fidelity. To address the computational cost of high-dimensional data, Latent Diffusion Models 34 and autoregressive methods 44 shift the generative process into a

Read the complete text