VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

Xuyang Chen, Guojian Wang, Keyu Yan, Lin Zhao · 2025

Read VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning by Xuyang Chen, Guojian Wang, Keyu Yan, Lin Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11944.txt.

Introduction

Xuyang Chen and co-authors present VIPO, a 2025 algorithm for learning from fixed datasets in reinforcement learning. Offline learning matters where live interaction is risky or costly, such as medicine or robotics. Model-based methods are data efficient, but model errors push them toward hand-tuned caution that can be unreliable.

VIPO instead penalizes inconsistency in the learned value function, using self-supervised feedback to spot trouble. Replacing heuristic uncertainty with a consistency signal should make the caution better aimed. The goal is policies that are careful exactly where the data cannot support confidence.

Catalog record

CREATORXuyang Chen, Guojian Wang, Keyu Yan, Lin Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11944

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning Xuyang Chen, Guojian Wang, Keyu Yan, Lin Zhao Abstract Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the one estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. It offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy. As a result, VIPO achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks. 1 Introduction Offline reinforcement learning (RL) ( Lange et al. 2012 ; Levine et al. 2020 ) aims to learn effective policies exclusively from a pre-collected behavior dataset, eliminating the need for online interaction with the environment. This approach offers a compelling solution for tasks where online interactions entail significant risks or exorbitant costs. Due to its potential to transform static datasets into powerful decision-making systems, offline RL has gained popularity in recent years ( Kalashnikov et al. 2018 ; Prudencio et al. 2023 ) . While off-policy RL algorithms can in principle be directly applied to the offline datasets, recent studies show that they can perform poorly when applied offline ( Fujimoto et al. 2019 ; Kumar et al. 2019 ) . This is primarily attributed to the distribution shift between the learned and the behavior policies, as learning policies beyond the behavior policy requires querying the actions not observed in the dataset. Similarly, value function evaluation on out-of-distribution (OOD) actions is usually inaccurate. In turn, maximizing an inaccurate value function during policy improvement often overestimates the value of OOD actions, which is a core challenge in offline RL. To mitigate overestimation, a common paradigm in offline RL is to incorporate conservatism into algorithm design. Model-free offline RL algorithms ( Fujimoto et al. 2018 ; Wu et al. 2019 ; Kumar et al. 2020 ; Wang et al. 2022 ) achieve this by learning a pessimistic value function (value-based approach) to discourage choosing OOD actions or constraining the learned policy (policy-based approach) to avoid visiting such actions. However, these algorithms often suffer from overly conservatism as many of them solely learn from the dataset ( Wang et al. 2018 ; Chen et al. 2020 ; Kostrikov et al. 2021 ) . Meanwhile, it is crucial to balance conservatism and generalization since being overly conservative hinders finding a better policy. In contrast, model-based offline RL algorithms ( Kidambi et al. 2020 ; Yu et al. 2020 ; Sun et al. 2023 ) ensure conservative by learning a pessimistic dynamics model which underestimates the value of OOD actions. Additionally, model-based algorithms have the potential for broader generalization as they can incorporate dynamics models to generate synthetic data that are not present in the dataset ( Yu et al. 2020 ) . This advantage makes model-based methods particularly well-suited for offline RL, motivating their further investigation in this paper. Figure 1: Illustration of our method (VIPO) and previous model-based approaches for learning a pessimistic dynamics model. A : Paradigm of model-based methods relying on uncertainty estimation. B : Paradigm of VIPO, where V_{d}^{\mu}(s) represents the value function learned from the data, and V_{m}^{\mu}(s) represents the value function learned from the model. Our method differs from prior work in two key aspects. First, we employ a dual-use strategy for the data, leveraging it to learn both a value function and a model. Second, we integrate the value inconsistency as a measure of model error, which is optimized during the model training process. Most prior model-based algorithms rely on heuristic uncertainty estimation to incorporate conservatism. They commonly train an ensemble of N models through maximum likelihood on the dataset and quantify uncertainty based on the standard deviation among the predictions of these N models. Specifically, Yu et al. 2020 employs the max-aleatoric uncertainty quantifier, Kidambi et al. 2020 adopts the max-pairwise-diff uncertainty quantifier, and Lu et al. 2021 utilizes the ensemble-standard-deviation uncertainty quantifier. Nevertheless, in practice, such heuristic uncertainty estimation methods could be challenging and unreliable when dealing with complex datasets or using deep neural network models ( Ovadia et al. 2019 ) . Moreover, we empirically demonstrate in Section 4.2 that even in Gym tasks, uncertainty can become invalid if the learned model is inaccurate. In this paper, we introduce VIPO ( V alue Function I nconsistency P enalized O ffline Reinforcement Learning), a model-based offline RL algorithm that aims to learn a highly accurate model by integrating a value function inconsistency loss into model training. VIPO differs fundamentally from prior methods in two key aspects. First, our approach employs the dataset in two distinct ways. As illustrated in Figure 1 , previous model-based methods use ensembles to estimate uncertainty, penalizing conflicting actions and favoring decisions that are consistent across the models. However, these ensemble models are trained using the same method, differing only in initialization, which reflects a single-purpose use of data. In contrast, VIPO employs a dual-use strategy for the data. First, an approximated value function for the behavior policy is trained by minimizing the empirical Mean Square Bellman Error (MSBE) using the dataset. Then, a model is learned by minimizing the negative log-likelihood loss on the same dataset, followed by value evaluation to obtain another approximated value function for the behavior policy. If the learned model is accurate, the value function derived from the model should be consistent with the value function learned from the dataset, owing to the uniqueness of the value function. This alignment—or lack thereof—serves as a quantifier of model accuracy. Note that the key to the success of offline RL algorithms lies in effectively leveraging the dataset, as it represents the entirety of available information. Therefore, compared with prior approaches that use data in a single-purpose manner, our method excels by employing a dual-use strategy and developing self-supervised training to enhance model learning. Second, unlike previous work which often disregards uncertainty, we instead try to minimize the value function inconsistency by incorporating it into the model training error, introducing an augmented model error. By establishing the analytical expression for the gradient of this augmented error in Theorem 3.2 , we enable the model to be trained via gradient descent, ensuring a balance between data fidelity and alignment in value functions. Our empirical results demonstrate from multiple perspectives that incorporating value function inconsistency into model error enhances model accuracy, highlighting the effectiveness of the aforementioned two novel designs. VIPO offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy. As a result, VIPO outperforms prior offline RL algorithms and achieves state-of-the-art

Read the complete text