Quantifying model prediction sensitivity to model-form uncertainty

Teresa Portone, Rebekah D. White, Joseph L. Hart · 2025

Read Quantifying model prediction sensitivity to model-form uncertainty by Teresa Portone, Rebekah D. White, Joseph L. Hart (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08708.txt.

Introduction

Teresa Portone and 2 co-authors present a method to quantify prediction sensitivity to model-form uncertainty in a 2025 paper. Model-form uncertainty comes from assumptions made when building a physics model. It is widely seen as important, but hard to measure beyond available data.

Parameterized changes, called MFU representations, stand in for altered assumptions. Grouped variance based sensitivity analysis then measures how much each assumption matters. The method works without calibration data, unlike many earlier approaches. If such data exist, they can shape the representation and handle dependence between parameters. Managers can better decide where to spend effort reducing model error.

Catalog record

CREATORTeresa Portone, Rebekah D. White, Joseph L. Hart
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantifying model prediction sensitivity to model-form uncertainty Teresa Portone, Rebekah D. White, Joseph L. Hart Abstract Model-form uncertainty (MFU) in assumptions made during physics-based model development is widely considered a significant source of uncertainty; however, there are limited approaches that can quantify MFU in predictions extrapolating beyond available data. As a result, it is challenging to know how important MFU is in practice, especially relative to other sources of uncertainty in a model, making it difficult to prioritize resources and efforts to drive down error in model predictions. To address these challenges, we present a novel method to quantify the importance of uncertainties associated with model assumptions. We combine parameterized modifications to assumptions (called MFU representations) with grouped variance-based sensitivity analysis to measure the importance of assumptions. We demonstrate how, in contrast to existing methods addressing MFU, our approach can be applied without access to calibration data. However, if calibration data is available, we demonstrate how it can be used to inform the MFU representation, and how variance-based sensitivity analysis can be meaningfully applied even in the presence of dependence between parameters (a common byproduct of calibration). 1 Introduction In high-consequence science and engineering problems it is often necessary to rely on physics-based mathematical models to predict the behavior of a complex system or phenomenon of interest, e.g., making predictions about the future or for conditions where experiments are impossible. However, in the process of model development, assumptions must be made that can degrade the trustworthiness of these models. Such assumptions include, but are not limited to, assumptions regarding invariance under transformation (e.g. rotation: isotropy, translation: homogeneity), couplings between phenomena, and dependencies (e.g., ignoring microscale dependence, assuming rate dependence). Uncertainty associated with the mathematical form and validity of these assumptions leads to model-form uncertainty (MFU) , where the exact mathematical representation needed to model a phenomenon of interest is unknown or cannot be represented using the information present in the model (e.g., dependence on quantities below the length-scale resolved in the model). MFU and other sources of uncertainty, such as parameter uncertainty, result in uncertainties in model predictions that must be quantified to adequately characterize the trustworthiness of resulting predictions. Being able to identify the most important sources of uncertainty is critical for determining where to invest resources to most improve prediction reliability through model improvement, uncertainty reduction, etc. To this end, this work presents a first-of-its-kind method to quantify the impact of modeling assumptions on model predictions relative to other sources of uncertainty. There are a variety of ways to represent uncertainty in a model’s representation of physics. The first established and most common approach was presented in [ 1 ] , wherein a statistical model is appended to the output of a physical model, and its hyperparameters are calibrated to minimize misfit with data. In [ 2 ] MFU is discussed and a process to evaluate it for a given data set is provided, but no guidance is provided on how such results could be interpolated (or extrapolated) beyond that data set. In [ 3 ] , MFU is characterized by augmenting uncertainty in a modeling assumption’s parameterization to account for discrepancy between data and predictions without perturbing the model assumption’s mathematical form. In [ 4 ] , a data-driven modification to modeling assumptions is used to represent MFU. In [ 5 , 6 , 7 ] a statistical model for the time evolution of a correction to the assumption is inferred using time-series data. All the aforementioned approaches rely on access to observational data, which is often unavailable in scenarios where model predictions are most needed, e.g., to inform policy and engineering design decisions. This work instead leverages what we term herein MFU representations , which characterize MFU through parameterized modifications to modeling assumptions [ 8 , 9 , 10 , 11 , 12 , 13 ] . Through their parameterizations, MFU representations are constrained to respect known phenomenology and relevant physics, such as conserved quantities, with the aim of maximizing their extrapolative capability. To represent uncertainty, their parameterizations are assigned probability distributions, where relevant physical constraints and phenomenological behavior may also be encoded. The resulting MFU representation provides a range of plausible behavior for the modeled phenomenon. Because they are developed to respect known physics and phenomenology, MFU representations can be developed without access to calibration data, which is a common situation in the preliminary stages of model development and uncertainty analysis. However, if data is available, MFU representations can be informed using, e.g., Bayesian inference. Hierarchical frameworks that calibrate the hyperparameters of an MFU parameterization are common, so we demonstrate one such a framework here. We use grouped variance-based sensitivity analysis (VBSA) to quantify the impact of physics modeling assumptions on model predictions herein. Namely, grouped Sobol’ indices measure the fraction of variance in a model prediction that can be attributed to each input factor in a model. Standard practice has focused on individual model parameters as input factors, but an input factor can also be defined as a group of parameters [ 14 , 15 , 16 ] . In the context of a group of parameters, Sobol’ indices measure the fraction of variance in a model prediction that can be attributed to all parameters in the group as well as to their interactions with each other. In this work, we leverage grouped Sobol’ indices to provide a single sensitivity measure for the group of parameters comprising an MFU representation, relative to other sources of uncertainty in the model. Since multiple MFU representations may be valid in a given modeling scenario, we establish bounds for the differences in Sobol’ indices corresponding to varying MFU representations. Furthermore, we demonstrate how grouped VBSA can be used to analyze sensitivity to MFU when hierarchical probability densities are used, even after calibration, which often induces dependence between MFU and other model parameters. While the proposed approach provides a measure of importance of modeling assumptions to model predictions, it can also measure the importance of model components/subcomponents whose parameterizations can be grouped in an uncertainty analysis. Thus, our proposed methodology can be used to inform where to invest resources, e.g., more experimentation or computation, to drive down the most impactful sources of uncertainty on model predictions. Furthermore, following the spirit of [ 17 , 18 ] , it can be used to assess whether assumptions important to prediction use cases are also important in validation cases as a means of assessing the relevance of validation evidence. The key contributions of this work include: \bullet Presenting a novel approach to measure the impact of model-form uncertainty on model predictions using variance-based sensitivity analysis. \bullet Proving that grouped variance-based sensitivity analysis is robust to an MFU representation’s parameterization. \bullet Analyzing computation of Sobol’ indices for hierarchical distributions. The remainder of the paper proceeds as follows. Section 2 presents the methodology of our proposed approach and provides the relevant background related to MFU representations ( Section 2.1 ) and their calibration ( Section 2.2 ) as well as grouped Sobol’ indices ( Section 2.3 ); an illustrative example is provided to build intuition. In Section 3.1 , we establish robustness of the approach to varying MFU representations. We demonstrate our approach on

Read the complete text