Conformal Prediction on Quantifying Uncertainty of Dynamic Systems

Aoming Liang, Qi Liu*, Lei Xu, Fahad Sohrab, Weicheng Cui, Changhui Song, Moncef Gabbouj* · 2024

Read Conformal Prediction on Quantifying Uncertainty of Dynamic Systems by Aoming Liang, Qi Liu*, Lei Xu, Fahad Sohrab, Weicheng Cui, Changhui Song, Moncef Gabbouj* (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10459.txt.

Introduction

Aoming Liang and co-authors bring conformal prediction to uncertainty scoring for learned models of moving physical systems, in a paper from two thousand twenty-four. Models that read physics from video need trust scores with real guarantees. Physical data uncertainty had lacked a systematic treatment.

The method wraps benchmark operator learners and returns ranges with stated coverage. Two rivals, dropout sampling and ensembles, were compared on equation data sets. Scores came from straight roll-outs of the forecast, suiting time-series work well. The authors frame the approach as a reliability layer for spatial intelligence systems.

Catalog record

CREATORAoming Liang, Qi Liu*, Lei Xu, Fahad Sohrab, Weicheng Cui, Changhui Song, Moncef Gabbouj*
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10459

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Conformal Prediction on Quantifying Uncertainty of Dynamic Systems Aoming Liang, Qi Liu*, Lei Xu, Fahad Sohrab, Weicheng Cui, Changhui Song, Moncef Gabbouj* Abstract Numerous studies have focused on learning and understanding the dynamics of physical systems from video data, such as spatial intelligence. Artificial intelligence requires quantitative assessments of the uncertainty of the model to ensure reliability. However, there is still a relative lack of systematic assessment of the uncertainties, particularly the uncertainties of the physical data. Our motivation is to introduce conformal prediction into the uncertainty assessment of dynamical systems, providing a method supported by theoretical guarantees. This paper uses the conformal prediction method to assess uncertainties with benchmark operator learning methods. We have also compared the Monte Carlo Dropout and Ensemble methods in the partial differential equations dataset, effectively evaluating uncertainty through straight roll-outs, making it ideal for time-series tasks. I Introduction Video data [ 1 , 2 , 3 ] can be represented by a set of time-dependent partial differential equations (PDEs), where the solutions correspond to the dynamic behavior. Similarly, image data can be interpreted as the solutions to time-independent partial or ordinary differential equations [ 4 ] . Exploring the uncertainties in PDEs using various methods is highly beneficial, particularly for tasks such as video understanding and generation [ 5 ] . PDEs serve as a fundamental framework for physical phenomena, with significant applications in weather forecasting [ 6 ] , nuclear fusion [ 7 ] , and molecular dynamics simulations [ 8 ] . Extensive research [ 9 , 10 , 11 ] has been undertaken on how to learn the PDEs by artificial intelligence, especially those incorporating temporal evolution data, which can be conceptualized as sequences of video data structured on uniform or non-uniform grids. Given the deterministic nature of physical laws, the evolution processes should be deterministic. The leading work primarily focuses on neural operator learning [ 12 ] . However, the inherent uncertainties in artificial intelligence models [ 13 ] pose challenges in quantifying the uncertainties during the learning processes and exploring whether calibration can mitigate these uncertainties. Uncertainty quantification provides insights into the credibility or confidence level of predictive outcomes. The researcher can better evaluate the model’s stability, accuracy, and reliability by accurately quantifying uncertainties. Classic uncertainty studies [ 14 , 15 , 16 ] only focus on one-step predictions of models, neglecting the cumulative uncertainties during temporal evolution in the forward or inverse problems. Rotational invariance [ 17 , 18 ] is a fundamental concept in the realm of physical systems, underpinning the understanding of symmetry in nature. Its significance is rooted in the principle that physical laws should remain unchanged under rotational transformations of the coordinate system. This invariance not only highlights the universality of physical laws but also plays a crucial role in the development of theoretical frameworks across various fields of physics. The main contributions of this work are as follows: • In this study, a complete quantitative assessment of uncertainty quantification methods is performed, including Monte Carlo dropout, conformal prediction, and ensemble. • The rotation symmetry in physical systems served as an opinion for evaluating uncertainty, utilizing calibration datasets to eliminate uncertainties is feasible and aligns with physical interpretation. • Through conformal prediction, our work enhances the interpretability of models for time series tasks and provides theoretical upper and lower bounds as guarantees. II Related Work on uncertainty evaluation Bayesian methods : Reference [ 19 ] firstly proposed introduced the Bayesian criterion, which assigns a prior distribution to the parameters of neural networks. With the training data, it then computes posterior distributions of these parameters to quantify predictive uncertainty. Reference [ 20 ] proposed the variational Bayesian inference in the neural network. In practice, the prior is often set to Gaussian distributions to convert the classical Bayesian to an optimization problem, and computing the posterior requires multiple iterations, with time-consuming in high-dimensional data. Monte Carlo dropout : Based on classic Bayesian, reference [ 21 ] suggested a Monte Carlo Dropout (MC Dropout) which acts as an approximation of Gaussian processes. With each forward pass, the network can randomly drop different nodes by utilizing a distinct random seed, resulting in diverse predictions for the same input, as depicted in Figure 1 (c). Ensemble : Ensemble methods, as shown in Figure 1 (a). Typically, models are assumed to be homoskedastic and exhibit the same variance distribution across all input data [ 22 ] . Researchers have recognized that performance improvements caused by assembling can lead to more stable predictions [ 23 , 24 ] . Conformal prediction (CP) : Conformal prediction [ 25 , 26 , 27 ] , as illustrated in Figure 1 (b), is a relatively novel method for assessing uncertainty. Fundamentally, it achieves this by transforming an algorithm’s predictions into prediction sets that guarantee robust coverage even for finite samples. Unlike traditional methods that might only assess the uncertainty at a single point, it leverages the information from the calibration dataset to ensure that the coverage probability of prediction intervals is not less than the chosen confidence level, regardless of the underlying distribution. This makes it particularly valuable in settings where accurate and reliable prediction intervals are crucial, such as in the issue of temporal dynamics in PDEs, where uncertainties can accumulate over time and symmetry. III Methods In this section, we provide a conceptual introduction to the foundational theory of conformal prediction, outline the PDEs tasks, and explain how the conformal prediction can be applied to problems in this study. Fig. 1: The model is evaluated using three methods: (a) the ensemble method, (b) conformal prediction, and (c) Monte Carlo dropout. III-A CP: Preliminary The core of CP is to construct a prediction set that provides uncertainty estimates for machine learning models. Let (X_{i},Y_{i})\sim P , i=1,\dots,n , represent n independent and identically distributed (i.i.d.) samples drawn from an unknown distribution P , where: • X_{i}\in\mathcal{X} represents the feature/input space, • Y_{i}\in\mathcal{Y} represents the response/output space, • \mathcal{X}\subseteq\mathbb{R}^{d} and \mathcal{Y}\subseteq\mathbb{R} , though this can be generalized to other domains. Define a set of prediction scores: \hat{C}_{n}:\mathcal{X}\to\{\text{subsets of }\mathcal{Y}\}, where \hat{C}_{n}(x) is the prediction set for a new input x\in\mathcal{X} . The function \hat{C}_{n} is constructed such that it satisfies the following finite-sample coverage property for any nominal error level \alpha\in(0,1) : \mathbb{P}\left(Y_{n+1}\in\hat{C}_{n}(X_{n+1})\mid(X_{i},Y_{i})_{i=1}^{n}\right)\geq 1-\alpha, where: • (X_{n+1},Y_{n+1})\sim P represents a new i.i.d. sample from the same distribution P , • the probability is computed over the randomness of the data (X_{i},Y_{i}) , i=1,\dots,n+1 . This property ensures that the constructed prediction sets \hat{C}_{n}(x) achieve valid coverage with probability at least 1-\alpha , even with finite training data. III-B Description of PDEs task Consider a time-dependent PDE of the form: \frac{\partial u}{\partial t}=\mathcal{F}(u,\nabla u,\nabla^{2}u,\dots), where u(t,x) is the state variable defined on a domain \Omega\subseteq\mathbb{R}^{d} , and \mathcal{F} is a differential operator governing the dynamics of u . The goal is to predict the state u_{n+1} at the next time step t_{n+1} , given the state u_{n} at the current time step t_{n} . Let (u_{n},u_{n+1})\sim P , where: • u_{n}\in\mathcal{U} is the state of the system at time t_{n} , • u_{n+1}\in\mathcal{U} is the state at time t_{n+1} , • P represents the underlying

Read the complete text