Video prediction using score-based conditional density estimation
Pierre-Étienne H. Fiquet & Eero P. Simoncelli · 2024
Read Video prediction using score-based conditional density estimation by Pierre-Étienne H. Fiquet & Eero P. Simoncelli (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00842.txt.
Description
A 2024 arXiv research paper by Pierre-Étienne H. Fiquet & Eero P. Simoncelli. Temporal prediction is inherently uncertain, but representing the ambiguity in natural image sequences is a challenging high-dimensional probabilistic inference problem. For natural scenes, the curse of dimensionality renders explicit density estimation statistically and computationally intractable. Here, we describe an implicit regression-based framework for learning and sampling the conditional density of the next frame in a video given previous observed frames.
Catalog record
| CREATOR | Pierre-Étienne H. Fiquet & Eero P. Simoncelli |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00842
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Video prediction using score-based conditional density estimation Pierre-Étienne H. Fiquet & Eero P. Simoncelli Abstract Temporal prediction is inherently uncertain, but representing the ambiguity in natural image sequences is a challenging high-dimensional probabilistic inference problem. For natural scenes, the curse of dimensionality renders explicit density estimation statistically and computationally intractable. Here, we describe an implicit regression-based framework for learning and sampling the conditional density of the next frame in a video given previous observed frames. We show that sequence-to-image deep networks trained on a simple resilience-to-noise objective function extract adaptive representations for temporal prediction. Synthetic experiments demonstrate that this score-based framework can handle occlusion boundaries: unlike classical methods that average over bifurcating temporal trajectories, it chooses among likely trajectories, selecting more probable options with higher frequency. Furthermore, analysis of networks trained on natural image sequences reveals that the representation automatically weights predictive evidence by its reliability, which is a hallmark of statistical inference 1 1 1 Code will be released upon acceptance. . 1 Introduction All organisms make temporal predictions, and more complex organisms make prediction about more complex aspects of their environment. Even in the simplest cases, temporal prediction is difficult: sensory measurements are incomplete and insufficient to fully specify the future, making prediction uncertain even at very short timescales. Consider a movie: the next frame of an image sequence is an event with several possible outcomes and a deterministic ( e.g. , least-squares) prediction computes an average over this multi-modal distribution, resulting in blurred predictions. Traditional image processing methods based on optic flow are prone to errors at occlusion boundaries, where motion is discontinuous and multiple next frames are possible. Point estimates are inadequate for handling such ambiguous situations and we consider a probabilistic formulation instead: video prediction amounts to estimating and sampling from the conditional density of the next frame given the previous frames in a sequence: p(x_{t+1}|x_{t},\dots,x_{t-\tau+1}) , where \tau is the memory length. Abstractly, temporal prediction can be thought of as an inverse problem that requires prior information in order to recover the part of the image sequence obfuscated by the arrow of time. Learning a density from data is generally considered intractable for high-dimensional signals such as videos: the sample complexity of density estimation is exponential in the signal dimensionality (the so-called “curse of dimensionality”). As a result, the statistical difficulty of high-dimensional density modeling can only be approached by making strong assumptions, and many methods have been proposed to tackle the case of video prediction ( Oprea et al. 2020 ) . Each of these methods imposes inductive biases through choices of objective function and/or prediction architecture. But, overall, video prediction remained largely out of reach for statistical machine learning methods until the advent of “diffusion models”, which have offered a new and highly successful paradigm for learning generative models ( Song & Ermon 2019 ; Ho et al. 2020 ; Song et al. 2021 ) . They have also been applied to video generation where they achieve impressive empirical results ( Ho et al. 2022 ; Brooks et al. 2024 ; Bar-Tal et al. 2024 ) . These results typically rely on huge datasets, opaque networks, and sophisticated text conditioning. In this paper, we will consider a simplified diffusion-like framework where the learned representation can be analyzed. Diffusion models are based on score-matching estimation and bypass the normalization constant that plagues maximum likelihood estimation ( Hyvärinen 2005 ) . This simplification arises from considering the gradient of the logarithm of the data distribution—known as the score function—which does not depend on the problematic normalization constant. Score-based estimation has its roots in the “empirical Bayes” subfield of statistical inference, and the central role of the score of the noisy data distribution can be generalized to many other types of estimation ( Raphan & Simoncelli 2011 ) . The empirical Bayes formulation is appealing because of its simplicity and generality: given data and a distortion process, least-squares regression gives access to the score of the noisy data. Traditional statistical machine learning frameworks such as variational autoencoders ( Kingma & Welling 2014 ) , or normalizing flows ( Dinh et al. 2014 ) , build an explicit model of the data distribution. In contrast, score-based modeling focuses on learning a mapping that approximates the score function and leaves the density model implicit. Remarkably, even vanilla convolutional deep networks can achieve excellent approximations of the optimal denoiser, thereby providing access to the score of the noisy data distribution, and enabling sampling from this distribution ( Kadkhodaie & Simoncelli 2021 ) . The similarities and differences of these two probabilistic modeling frameworks are illustrated in Figure 1 . Crucially, the score-based modeling framework offers the possibility of analyzing the representation, and here, we aim to describe how trained networks adapt to the content of the image being processed. Figure 1: Modeling frameworks for probabilistic prediction. Both the classical and the proposed approach are trained unsupervised on image sequences and learn the distribution of the next frame conditioned on the recent past. Both approaches make predictions by sampling from this learned conditional density, but the density can be either explicit or implicit. Left. Traditional framework: the model is explicit and its parameters are learned through repeated iterations of inference and sampling. The learning objective is applied end-to-end: from past frames, to inferred latent representation, to generated next frame. Right . Proposed score-based framework: train a denoiser via regression, i.e. , a mapping from past frames and noisy observation to an estimate of the clean next frame. The trained denoiser approximates the score function and implicitly represents the probabilistic model (denoted by a funnel). To sample from this implicit model, iterative partial denoising gradually transforms noise into a predicted next frame. This work proposes a simple probabilistic approach to video prediction that enables analysis of the learned representation, making three contributions: in Section 2 , we describe a score-based framework for conditional density estimation applied to video prediction; then in Section 3.1 , we analyze a representative case of bifurcating trajectory that arises due to occlusion boundaries; finally in Section 3.2 , we reveal the adaptivity of representations in trained networks, making their performance partially interpretable. 2 Score-based estimation and sampling of conditional densities Given access to a video dataset, i.e. , image sequences \{x_{t}\in\mathbb{R}^{d},t=1,\dots,T\} , we want to predict future frames on new videos, i.e. , generate probable next-frames on test image sequences. To achieve this, we describe a scalable framework for learning to sample from the distribution of the next frame conditioned on the recent past. First, we assume that the temporal dependencies are local and define a corresponding network architecture. Then, we introduce a useful intermediate object, the score of the noisy observation distribution, and propose a regression objective for training a network to approximate the score. Finally, we express an iterative procedure that utilizes the trained network for sampling probable next-frames. These methods extend previous work on unconditional density modeling ( Kadkhodaie & Simoncelli 2021 ) to the conditional setting of video prediction. An intuitive example with low-dimensional visualizations is given in Appendix B . 2.1 Conditioning with a sequence-to-image network architecture We assume that temporal dependencies are local and well approximated by the Markov