Conditional Deep Generative Models for Belief State Planning

Anthony Corso, Antoine Bigeard, Mykel J. Kochenderfer · 2025

Read Conditional Deep Generative Models for Belief State Planning by Anthony Corso, Antoine Bigeard, Mykel J. Kochenderfer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11698.txt.

Introduction

Anthony Corso, Antoine Bigeard, and Mykel Kochenderfer tackle planning for agents that cannot fully see the world around them. In such problems the agent keeps a belief, a spread of guesses about the true state. For rich tasks that belief is hard to store or update with older tools.

Their idea is to let a deep generative model hold the belief instead. The model learns from random trial runs and can then draw as many fresh samples from the belief as planning needs. On a mineral exploration task with a large, continuous state space, the approach beat particle filter methods. It matched the true belief better and made better decisions, which suggests it can scale to harder problems.

Catalog record

CREATORAnthony Corso, Antoine Bigeard, Mykel J. Kochenderfer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Conditional Deep Generative Models for Belief State Planning Anthony Corso, Antoine Bigeard, Mykel J. Kochenderfer Abstract Partially observable Markov decision processes (POMDPs) are used to model a wide range of applications, including robotics, autonomous vehicles, and subsurface problems. However, accurately representing the belief is difficult for POMDPs with high-dimensional states. In this paper, we propose a novel approach that uses conditional deep generative models (cDGMs) to represent the belief. Unlike traditional belief representations, cDGMs are well-suited for high-dimensional states and large numbers of observations, and they can generate an arbitrary number of samples from the posterior belief. We train the cDGMs on data produced by random rollout trajectories and show their effectiveness in solving a mineral exploration POMDP with a large and continuous state space. The cDGMs outperform particle filter baselines in both task-agnostic measures of belief accuracy as well as in planning performance. 1 Introduction Partially observable Markov decision processes (POMDPs) can model a wide range of applications such as robotics [ Pineau and Gordon 2007 ] , autonomous vehicles [ Wray et al. 2021 ] , and, more recently, subsurface problems such as ground water [ Wang et al. 2022 ] , carbon capture and storage [ Corso et al. 2022 ] , and mineral exploration [ Mern and Caers 2023 ] . A major challenge for solving POMDPs with high dimensional states is the difficulty of accurately representing the belief, which is the probability distribution over the state of the system given the history of actions and observations. In this paper, we propose a novel approach to address this challenge using conditional deep generative models (cDGMs) to represent the belief. Traditional belief representations, such as particle filters, are often not suitable for high-dimensional states and large numbers of observations. They often suffer from particle depletion , where, after successive iterations of re-weighting, only a small number of particles retain most of the weight, leading to a low quality approximation of the posterior belief (depicted in fig. 1(a) ). Our approach (depicted in fig. 1(b) ) trains a deep generative model that is conditioned directly on a sequence of actions and observations, and can generate an arbitrary number of samples from the posterior belief. The cDGMs are trained from data produced by random rollout trajectories, which allows for the complex Bayesian inversion to be learned directly from the training data. \ldots a_{0},o_{0} a_{1},o_{1} a_{T},o_{T} (a) Particle Filter Belief Update (a_{0},o_{0},\ldots a_{T},o_{T} ) cDGM (b) cDGM Belief Update Figure 1 : Summary of proposed approach We apply our proposed approach to solve a mineral exploration POMDP [ Mern and Caers 2023 ] that has a large and continuous state space as well as continuous, noise-free observations. We investigate two types of cDGMs: generative adversarial networks (GANs) and denoising diffusion probabilistic models (DDPMs). We find that both model classes outperform a particle filter baseline on task-agnostic metrics, and that the DDPM models enable significanlty improved planning performance. Prior work [ Mosser et al. 2018 , Laloy et al. 2017 , Mosser et al. 2017 , Song et al. 2021 ] has investigated the use of deep generative models for geoscience applications, but does not condition the generation of samples on observed data. To condition to observations, traditional Bayesian inversion approaches are combined with unconditioned deep generative models. Laloy et al. 2017 use MCMC with the generator of a variational autoencoder while Liu et al. 2019 use a form of principle component analysis to produce conditioned samples and use deep style transfer to refine the samples. In contrast to prior work, we condition to observations directly in the generative model, and use the model for POMDP planning. In summary, our contributions in this paper are as follows • We propose the use of conditional deep generative models to represent beliefs for belief-state planning solvers of POMDPs. • We experiment with a wide range of state-of-the-art deep generative models, and identity architectures that are well suited for observation conditioning. • We show that cDGM belief representations outperform other representations when planning under uncertainty in a mineral exploration problem. 2 Background This section introduces the topics of POMDPs, particle filter approximations to Bayesian inference and deep generative models. Then, we provide a summary of our motivating application domain: mineral exploration. 2.1 POMDPs A partially observable Markov Decision Process (POMDP) is a model for sequential decision making under state uncertainty [ Kochenderfer et al. 2022 ] . A POMDP is defined by the tuple ( S , A , T , R , O , Z , \gamma ) where S is the set of states, A is the set of actions, T is a transition model, R is the reward function, O is the observation space, Z is the observation model, and \gamma is the discount factor. At each timestep, an agent takes an action a\in A in a state s and transitions to the next state s^{\prime}\sim T(\cdot\mid s,a) , receiving a reward r=R(s,a) and an observation o\sim Z(\cdot\mid s,a,s^{\prime}) . Since states are not fully observed, the agent maintains a probability distribution over states or belief b(s) that is updated based on the action-observation trajectory \tau=\{a_{1},o_{1},a_{2},o_{2},\ldots\} . The agent’s policy a=\pi(b) maps beliefs to actions with the goal of maximizing the expected discounted sum of future rewards. POMDPs can be solved exactly only for problems with small state, action, and observation spaces, but for larger problems, approximate solvers are used [ Kochenderfer et al. 2022 ] . The most scalable approaches often use forward tree search, where hypothetical trajectories are simulated and used to estimate the value of different actions. Some examples of tree-search-based POMDP solvers include POMCP [ Silver and Veness 2010 ] , POMCPOW and PFT [ Sunberg and Kochenderfer 2018 ] . 2.2 Particle Approximations for Bayesian Inference Accurate belief representations are an essential element to solving POMDPs, both for planning and execution. After taking action a , the prior belief b(s) is updated with the observation o using Bayes’ rule b(s^{\prime})=O(o\mid a,s^{\prime})\int_{s}T(s^{\prime}\mid s,a)b(s)ds (1) Exact Bayesian inference is often intractable for nonlinear dynamics, and beliefs that are not well represented by simple parametric models. Instead, the belief can be approximated by a weighted set of state samples, or particles. The weight of each particle is computed as the conditional likelihood of the state give the action-observation trajectory w=p(s\mid\tau) . To avoid retaining \tau , implementations often use a bootstrap filtering algorithm where the belief is represented by N equally weighted samples of the state b\approx\{s_{i}\}_{i=1}^{N} that are updated according to the following computational steps: 1. Sample new states from the transition function: s_{i}^{\prime}\sim T(\cdot\mid s_{i},a) . 2. Compute particle weights according to the observation model: w_{i}=O(o\mid a,s_{i}^{\prime}) . 3. Resample N particles from \{s_{i}^{\prime}\}_{i=1}^{N} according to their weights \{w_{i}\}_{i=1}^{N} . Particle filters are flexible belief representations which have favorable convergence properties [ Crisan and Doucet 2002 ] , but they require the observation likelihood O(\cdot\mid,a,s^{\prime}) . In problems where the observation likelihood is undefined (as with noiseless observations), or intractable to compute, an additional approximation is needed to compute the likelihood weights. Approximate Bayesian computation (ABC) is a particle filtering approach where the likelihood function is replaced with a approximation \tilde{O} . For example, in the case

Read the complete text