Neural ODE and SDE Models for Adaptation and Planning in Model-Based Reinforcement Learning
Chao Han [email protected] of Computer Science, The University of Sheffield, UKPresent address: Cancer Research UK National Biomarker Centre, The University of Manchester, UK, Stefanos Ioannou [email protected] of Computer Science, The University of Sheffield, UK Luca Manneschi [email protected] of Computer Science, The University of Sheffield, UK T.J. Hayward [email protected] of Chemical, Materials and Biological Engineering, The University of Sheffield, UK Michael Mangan [email protected] of Computer Science, The University of Sheffield, UK Aditya Gilra [email protected] Learning group, Centrum Wiskunde & Informatica, Amsterdam, NetherlandsSchool of Computer Science, The University of Sheffield, UKPresent address: Institute for Ecological Economics, Vienna University of Economics and Business, Austria Eleni Vasilaki [email protected] of Computer Science, The University of Sheffield, UK · 2026
Read Neural ODE and SDE Models for Adaptation and Planning in Model-Based Reinforcement Learning by Chao Han [email protected] of Computer Science, The University of Sheffield, UKPresent address: Cancer Research UK National Biomarker Centre, The University of Manchester, UK, Stefanos Ioannou [email protected] of Computer Science, The University of Sheffield, UK Luca Manneschi [email protected] of Computer Science, The University of Sheffield, UK T.J. Hayward [email protected] of Chemical, Materials and Biological Engineering, The University of Sheffield, UK Michael Mangan [email protected] of Computer Science, The University of Sheffield, UK Aditya Gilra [email protected] Learning group, Centrum Wiskunde & Informatica, Amsterdam, NetherlandsSchool of Computer Science, The University of Sheffield, UKPresent address: Institute for Ecological Economics, Vienna University of Economics and Business, Austria Eleni Vasilaki [email protected] of Computer Science, The University of Sheffield, UK (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23245.txt.
Introduction
Han and six co-authors test neural differential equation models for model-based reinforcement learning. Neural stochastic equations capture random transition dynamics better than plain neural ODEs. That edge yields strong policies with fewer samples in hard settings. The team also adapts policies to changed dynamics through inverse models, using few new interactions. For partly observed tasks, they build a latent model that mixes an ODE with a GAN-trained random part. Policies from that model match or beat leading model-based and model-free baselines.
Catalog record
| CREATOR | Chao Han [email protected] of Computer Science, The University of Sheffield, UKPresent address: Cancer Research UK National Biomarker Centre, The University of Manchester, UK, Stefanos Ioannou [email protected] of Computer Science, The University of Sheffield, UK Luca Manneschi [email protected] of Computer Science, The University of Sheffield, UK T.J. Hayward [email protected] of Chemical, Materials and Biological Engineering, The University of Sheffield, UK Michael Mangan [email protected] of Computer Science, The University of Sheffield, UK Aditya Gilra [email protected] Learning group, Centrum Wiskunde & Informatica, Amsterdam, NetherlandsSchool of Computer Science, The University of Sheffield, UKPresent address: Institute for Ecological Economics, Vienna University of Economics and Business, Austria Eleni Vasilaki [email protected] of Computer Science, The University of Sheffield, UK |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59028 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23245
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural ODE and SDE Models for Adaptation and Planning in Model-Based Reinforcement Learning Chao Han [email protected] of Computer Science, The University of Sheffield, UKPresent address: Cancer Research UK National Biomarker Centre, The University of Manchester, UK, Stefanos Ioannou [email protected] of Computer Science, The University of Sheffield, UK Luca Manneschi [email protected] of Computer Science, The University of Sheffield, UK T.J. Hayward [email protected] of Chemical, Materials and Biological Engineering, The University of Sheffield, UK Michael Mangan [email protected] of Computer Science, The University of Sheffield, UK Aditya Gilra [email protected] Learning group, Centrum Wiskunde & Informatica, Amsterdam, NetherlandsSchool of Computer Science, The University of Sheffield, UKPresent address: Institute for Ecological Economics, Vienna University of Economics and Business, Austria Eleni Vasilaki [email protected] of Computer Science, The University of Sheffield, UK Abstract We investigate neural ordinary and stochastic differential equations (neural ODEs and SDEs) to model stochastic dynamics in fully and partially observed environments within a model-based reinforcement learning (RL) framework. Through a sequence of simulations, we show that neural SDEs more effectively capture transition dynamics’ inherent stochasticity, enabling high-performing policies with improved sample efficiency in challenging scenarios. We leverage neural ODEs and SDEs for efficient policy adaptation to changes in environment dynamics via inverse models, requiring only limited interactions with the new environment. To address partial observability, we introduce a latent SDE model that combines an ODE and a GAN-trained stochastic component in latent space. Policies derived from this model offer a strong baseline, outperforming or matching general model-based and model-free approaches across stochastic continuous-control benchmarks. This work illustrates the applicability of action-conditional latent SDEs for RL planning in environments with stochastic transitions. Our code is available at: https://github.com/ChaoHan-UoS/NeuralRL . † Equal Contribution. 1 Introduction In recent years, the family of neural differential equations (neural DEs) ( 3 ; 22 ; 19 ; 15 ; 14 ) have emerged as a powerful framework for modelling dynamical systems. The general idea of these models is to use neural networks to parameterise the derivatives of the system’s dynamics, while the state evolution is computed by a numerical differential equation solver. Such a way of decoupling the modelling of dynamics from the discretisation scheme leads to models with increased fidelity for capturing complex, continuous-time transitions. As a result, neural DEs are especially well-suited for learning and representing transition dynamics in reinforcement learning (RL). Recent work has demonstrated that integrating neural ordinary differential equations (neural ODEs) ( 3 ) and stochastic differential equations (neural SDEs) 19 ; 14 into RL frameworks can improve the modelling of continuous-time dynamics. In partially observable Markov decision processes (POMDPs) and model-free RL, recurrent neural ODEs have shown robustness to irregularly sampled observations, due to their capacity to model non-uniform time series ( 26 ) . Latent neural ODEs have also been employed as data-driven dynamics models in model-based RL, providing higher sample efficiency compared to model-free baselines ( 8 ) . Additionally, incorporating neural ODEs with control-theoretic constraints has supported the development of safe and stable RL in continuous-time domains ( 25 ) . Neural SDEs extend the neural ODE framework by jointly modelling the deterministic and stochastic components of system dynamics, allowing for uncertainty-aware policy learning. Recent studies show that offline model-based RL with neural SDEs, particularly when incorporating physics priors, can outperform state-of-the-art algorithms on low-quality datasets ( 18 ) . Physics-constrained neural SDEs, which explicitly represent model uncertainty, have also been shown to support generalisation beyond the training distribution ( 7 ) . While both these neural SDE frameworks could, in principle, be applied to POMDPs, practical demonstrations to date have focused on environments with full observability. Despite recent advances, the application of latent neural SDEs to partially observed, stochastic RL environments remains underexplored. Unified frameworks that combine controlled latent neural ODEs and neural SDEs in both fully and partially observed environments are lacking, and the practical benefits of modelling stochasticity in latent dynamics for planning have not been systematically assessed. Additionally, there has been limited exploration of neural ODEs and neural SDEs in settings that require adaptation to changes in the environment. In this work, we first demonstrate sample-efficient policy adaptation to changes in environment configuration by employing an inverse dynamics approach ( 5 ) based on neural ODE and SDE transition models. We then introduce a unified latent ODE/SDE framework for model-based RL, motivated by the need to handle partial observability in complex, stochastic environments. The framework uses a two-phase procedure: the mean latent dynamics are learned using a latent ODE, and stochasticity is handled via a GAN-trained latent SDE. In experiments on stochastic continuous control tasks, we show that model-based RL with latent neural SDEs improves sample efficiency compared to both ODE-based models and a model-free SAC baseline in the most challenging fully or partially observed environment included in our study. To our knowledge, there has been no prior empirical validation of action-conditioned latent SDEs for planning in partially observed stochastic control tasks. Our results highlight the potential benefit of latent neural SDEs for flexible, noise-aware planning in stochastic domains. 2 Background Markov decision process (MDP) and partially observable MDP (POMDP). An MDP ( 2 ) is defined by a tuple \langle\mathcal{S},\mathcal{A},\mathcal{T},R\rangle , where \mathcal{S} and \mathcal{A} are sets of states and actions respectively, \mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S}) is the probabilistic transition function (dynamics), with \mathcal{T}(s_{t+1}|s_{t},a_{t}) representing the probability of transitioning into s_{t+1} from s_{t} under a_{t} , R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R} is the deterministic reward function. The initial state s_{0} follows certain distribution \rho_{0} , the horizon is T\in\mathbb{N}^{+}\cup\{+\infty\} and the discount factor is \gamma\in[0,1] , with 1 for finite horizon. When the state s_{t} is not fully observable, MDP tuples are extended to POMDP ( 2 ) \langle\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{T},O,R\rangle by adding a set of observations \mathcal{O} and an emission function O:\mathcal{S}\rightarrow\Delta(\mathcal{O}) , which probabilistically maps a state s_{t} to an observation o_{t} . Specifically, we consider POMDPs with two sources of uncertainty in the transition dynamics of observed states \mathcal{T}(o_{t+1}\mid o_{t},a_{t}) : (i) aleatoric stochasticity in the underlying MDP transition \mathcal{T}(s_{t+1}\mid s_{t},a_{t}) that produces the observations, and (ii) epistemic uncertainty from partial observability, which can often be reduced by conditioning on a history of observations and actions. To focus on transition stochasticity and decouple it from aleatoric uncertainty in observation emission, we assume deterministic emission functions O , i.e., o_{t}=o(s_{t}) . Nonetheless, our approach readily generalizes to noisy observation emissions by replacing the deterministic observation model with a probabilistic neural network that outputs the statistics of the predicted observation distribution. In such POMDPs, we introduce a latent representation z_{t} that summarizes the observed history up to t \{a_{0:t-1},o_{0:t}\} as a proxy of the state s_{t} ( 8 ; 21 ) . The transition function of the latent z_{t} is \mathcal{T}(z_{t+1}|z_{t},a_{t},o_{t}) , with the observation o_{t} deterministically emitted from z_{t} via o_{t}=o(z_{t}) . We are interested in learning the encoder mapping the history to the latent variable, the latent transition function and the decoder mapping the latent back to the observation, which together enable the prediction of the next observation in the POMDP. Neural Ordinary Differential Equation (Neural ODE). Neural differential equations are a family of differential equations whose rate functions are approximated by learnable neural networks ( 16 ) . A classic example is a neural ODE ( 3 ) : \displaystyle\frac{\text{d}s_{t}}{\text{d}t}=f_{\theta}(s_{t},t),\quad\text{where}\quad