Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learning

Zeyu Fang, Zuyuan Zhang, Mahdi Imani, Tian Lan · 2026

Read Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learning by Zeyu Fang, Zuyuan Zhang, Mahdi Imani, Tian Lan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02900.txt.

Introduction

Zeyu Fang and 3 co-authors tackle a weakness of learning from fixed datasets in a 2026 paper. In offline reinforcement learning, an agent learns from stored experience without new trial and error. Improvement pushes it toward actions the dataset barely covers, where model errors grow and value estimates inflate.

Their method, MC-ETM, learns an energy-based model of next states constrained to a learned manifold. Hard negative examples near that manifold sharpen the model's sense of what is plausible. During learning, rollouts stop when energy signals unreliability. Value backups also get pessimistic penalties. Tests on standard offline control benchmarks follow. The method improved multi-step accuracy and returns, especially with sparse or irregular data.

Catalog record

CREATORZeyu Fang, Zuyuan Zhang, Mahdi Imani, Tian Lan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learning Zeyu Fang, Zuyuan Zhang, Mahdi Imani, Tian Lan Abstract Model-based offline reinforcement learning is brittle under distribution shift: policy improvement drives rollouts into state–action regions weakly supported by the dataset, where compounding model error yields severe value overestimation. We propose Manifold-Constrained Energy-based Transition Models (MC-ETM), which train conditional energy-based transition models using a manifold projection–diffusion negative sampler. MC-ETM learns a latent manifold of next states and generates near-manifold hard negatives by perturbing latent codes and running Langevin dynamics in latent space with the learned conditional energy, sharpening the energy landscape around the dataset support and improving sensitivity to subtle out-of-distribution deviations. For policy optimization, the learned energy provides a single reliability signal: rollouts are truncated when the minimum energy over sampled next states exceeds a threshold, and Bellman backups are stabilized via pessimistic penalties based on Q -value-level dispersion across energy-guided samples. We formalize MC-ETM through a hybrid pessimistic MDP formulation and derive a conservative performance bound separating in-support evaluation error from truncation risk. Empirically, MC-ETM improves multi-step dynamics fidelity and yields higher normalized returns on standard offline control benchmarks, particularly under irregular dynamics and sparse data coverage. 1 Introduction Offline reinforcement learning (RL) learns a policy solely from a fixed dataset collected by one or more behavior policies, without further interaction with the environment ( 20 ) . A fundamental challenge is distribution shift : as policy optimization progresses, the learned policy queries state–action pairs that are weakly supported by the dataset, where value estimation is poorly constrained and can become severely over-optimistic, leading to unstable training or performance collapse ( 9 ) . A common response is to suppress out-of-distribution (OOD) actions or constrain the learned policy toward the behavior distribution ( 8 ; 25 ; 17 ) . While effective at limiting extrapolation, such constraints can be overly conservative when the dataset is suboptimal. Model-based offline RL instead learns a transition model from trajectories and performs policy evaluation and improvement through synthetic rollouts ( 29 ; 16 ; 23 ) . However, this shifts the difficulty to robust transition modeling under shift : once rollouts approach the boundary of dataset support, even small transition errors can compound rapidly and drive value targets toward unrealistic, high-return trajectories ( 11 ) . This brittleness is amplified in realistic environments for two structural reasons. First, many systems exhibit multi-modal or discontinuous transitions (e.g., mode switching, bifurcations, contact-like effects), where regression-style forward models trained with mean squared error average incompatible outcomes and yield implausible predictions ( 19 ; 1 ) . Second, although observations may be high-dimensional, valid next states often concentrate near low-intrinsic-dimensional structures; perturbations that are small in ambient space can correspond to large departures from the transition support, destabilizing long-horizon evaluation even near the data boundary ( 5 ; 21 ) . We propose Manifold-Constrained Energy-based Transition Models (MC-ETM) for robust offline model-based RL. MC-ETM trains conditional energy-based transition models with a geometry-aware manifold projection–diffusion (MPD) mechanism ( 26 ) : it learns a latent manifold of next states and generates near-manifold hard negatives by perturbing latent codes of observed transitions and running Langevin dynamics under the learned energy. Concentrating negatives near the dataset support while enforcing conditional inconsistency forces the energy to form sharper contours around valid transitions, improving sensitivity to small but consequential OOD deviations. MC-ETM then uses the learned conditional energy as a single reliability signal for offline policy optimization. We truncate synthetic rollouts when the inferred minimum energy over sampled next states exceeds a threshold, and we stabilize Bellman backups using pessimistic penalties based on Q -value-level dispersion across energy-guided samples. We formalize the resulting algorithm as policy optimization in a hybrid pessimistic Markov Decision Process (MDP) and derive a conservative performance bound that separates in-support evaluation error from truncation risk. Our main contributions are: • Geometry-aware energy-based transition learning: We introduce MC-ETM with MPD, which generates near-manifold hard negatives via latent perturbation and Langevin dynamics to sharpen the conditional energy landscape near the dataset support. • Reliability-aware offline policy optimization: We use the learned energy to control rollout reliability through minimum-energy truncation and to stabilize learning via pessimistic Q -value penalties computed from energy-guided samples. • Theory and evidence: We provide a hybrid pessimistic MDP formulation with a conservative bound separating in-support evaluation error from truncation risks, and demonstrate improved dynamics fidelity and higher normalized returns on standard offline control benchmarks (including D4RL MuJoCo), particularly under irregular dynamics and sparse data coverage. 2 Related Work Model-free offline RL. A large body of work mitigates distribution shift without an explicit dynamics model by constraining policy improvement toward the dataset support via behavior regularization or conservative value estimation (e.g., BCQ/BEAR/CQL/IQL/TD3+BC/EDAC and related variants) ( 9 ; 14 ; 15 ; 13 ; 8 ; 2 ; 3 ; 31 ) . These methods improve robustness by limiting extrapolation on OOD samples and value targets, but they can become overly conservative when the dataset is suboptimal and improvement requires generalization. Model-based offline RL. Model-based offline RL learns a transition model from data and performs policy evaluation and improvement via synthetic rollouts, typically coupled with pessimism to reduce exploitation of model error under rollout shift (e.g., MOPO/MOReL/COMBO/MOBILE and related frameworks) ( 29 ; 12 ; 28 ; 22 ; 23 ; 30 ; 27 ) . Performance hinges on rollout reliability near the dataset boundary, where small transition errors compound and common uncertainty surrogates (often next-state disagreement/variance) can be misaligned with decision-relevant value overestimation in high-dimensional settings. Energy-based transition modeling. Energy-based transition models (ETMs) represent conditional dynamics via a scalar energy over candidates and perform inference by sampling, capturing multi-modal or discontinuous transitions without collapsing to a single averaged prediction ( 6 ; 4 ) . Because ETMs are trained contrastively, the negative-sample distribution is central: in high dimensions, noise-initialized negatives are often far from the dataset support, yielding “easy” contrasts that under-shape the energy landscape precisely in the near-support regime where subtle deviations govern rollout reliability. ( 18 ; 24 ) Our approach. Our approach makes energy-based transition learning geometry-aware for offline model-based RL by introducing an MPD mechanism that learns a latent manifold of next states and generates near-manifold hard negatives via latent perturbations and Langevin dynamics under the learned conditional energy, sharpening the energy boundary near the dataset support. The learned energy then serves as a unified reliability signal for offline policy optimization, enabling minimum-energy rollout truncation and value-level pessimistic regularization based on Q -dispersion across energy-guided samples. 3 Preliminaries General RL The goal of reinforcement learning is to learn a policy that maximizes the expected return of an unknown specific Markov Decision Process (MDP), referred as the environment. A standard discounted MDP denoted as \mathcal{M} can be defined as a six-element tuple \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma,\mu_{0}), where \mathcal{S} is the state space, \mathcal{A} is the action space, P(s^{\prime}\mid s,a) is the state transition probability distribution, in which s,s^{\prime}\in\mathcal{S} and a\in\mathcal{A} . r(s,a)\in\mathbb{R} is the reward function, \gamma\in(0,1) is the discount factor, and \mu_{0} is the initial distribution of states. For a given policy \pi(a|s) , it can iteratively interact with this MDP in discrete time steps and induce a trajectory (s_{0},a_{0},r_{0},s_{1},\dots) . Based on \pi , the value

Read the complete text