A Sensitivity Analysis Framework for Quantifying Confidence in Decisions in the Presence of Data Uncertainty

Adway S. Wadekar · 2025

Read A Sensitivity Analysis Framework for Quantifying Confidence in Decisions in the Presence of Data Uncertainty by Adway S. Wadekar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17043.txt.

Introduction

Policy choices rest on data that are flawed by error, gaps, and bias, and the way analysts patch those flaws can sway the outcome. Adway Wadekar and co-authors propose a 2025 framework for testing how much a decision should be trusted anyway.

The idea is to define scores and plots that target the final decision, not just the raw estimates. Analysts vary their patching assumptions and watch whether the decision itself would flip. Two worked examples show the method: one weighs measurement error in a model of vote share, the other weighs filled-in values when judging how many children face high lead levels. The framework aims to make confidence in a policy call explicit and checkable.

Catalog record

CREATORAdway S. Wadekar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-17043

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Sensitivity Analysis Framework for Quantifying Confidence in Decisions in the Presence of Data Uncertainty Adway S. Wadekar Abstract Nearly all statistical analyses that inform policy-making are based on imperfect data. As examples, the data may suffer from measurement errors, missing values, sample selection bias, or record linkage errors. Analysts have to decide how to handle such data imperfections, e.g., analyze only the complete cases or impute values for the missing items via some posited model. Their choices can influence estimates and hence, ultimately, policy decisions. Thus, it is prudent for analysts to evaluate the sensitivity of estimates and policy decisions to the assumptions underlying their choices. To facilitate this goal, we propose that analysts define metrics and visualizations that target the sensitivity of the ultimate decision to the assumptions underlying their approach to handling the data imperfections. Using these visualizations, the analyst can assess their confidence in the policy decision under their chosen analysis. We illustrate metrics and corresponding visualizations with two examples, namely considering possible measurement error in the inputs of predictive models of presidential vote share and imputing missing values when evaluating the percentage of children exposed to high levels of lead. Keywords: election, imputation, measurement, missing, nonignorable. 1 Introduction There is no question that statistical analyses can inform policy-making. Often, however, the data used for such analyses are messy and imperfect. Values of important variables may be missing or measured with error. Data files constructed by linking records across multiple databases may be prone to incorrect or incomplete linkages. The data themselves may be collected from a sample that is not representative of the full population. Data analysts must confront such imperfections as part of the analysis process. This could involve explicitly accounting for data imperfections. For example, they may decide to impute missing values according to some (possibly missing not at random) model and subsequently analyze the completed datasets ( Little & Rubin 2019 ) . Or, they may decide to embed a measurement error model in their analysis ( Fuller 1987 ) . Alternatively, they could simply disregard the imperfections and treat the collected data “as is” in the analysis. For example, they could analyze only the complete cases or fix possibly erroneous values at their recorded values. Even with such do-nothing choices, the validity of the resultant analyses still relies on implied assumptions, e.g., the data are missing or faulty completely at random ( Reiter & Raghunathan 2007 ) . For some policy questions, analysts utilize data provided by a statistical agency. Frequently, such data are preprocessed by the agency, which corrects imperfections as best it can. This too represents a modeling choice: the analyst implicitly accepts the assumptions underpinning the agency’s preprocessing steps ( Meng 2014 ) . The analyst’s choices in dealing with imperfections can influence their statistical inferences. In turn, this ultimately can affect policy decisions that derive from those inferences. It is therefore prudent for analysts to assess the sensitivity of statistical inferences, and arguably more importantly the ultimate decisions that stem from those inferences, to the reasonableness of the assumptions underpinning their approach for dealing with data imperfections. In this article, we present a framework to facilitate such sensitivity analyses. The key idea is to define a metric that targets the sensitivity of the ultimate decision to the assumptions underlying the approach to handling the data imperfections. We emphasize metrics for decisions as opposed to statistical inferences, as ultimately it is the decision that matters most. To illustrate, suppose a health agency seeks to estimate the percentage of people exposed to a communicable disease. It may not matter if data imperfections cause the estimate to be off by, say, 5% when the true percentage is high; whether the estimate is 75% or 80%, the conclusion (decision) is that the disease has spread to a sizable fraction of the population. On the other hand, in a different context, say political polling, an error of 5% could lead to a grossly incorrect decision. In these examples, the important metric is not the amount of error in an absolute sense—that is, purely statistical concepts like bias, mean-squared error, and confidence interval coverage—but the amount of error given the context of the decisions that will be made from the analysis. After defining a metric, analysts compute it under their proposed method for handling the data imperfections. They also compute it under varying degrees of departure from the assumptions underlying their proposed method. In this way, analysts can assess how much of a departure is needed before the decision changes in practically meaningful ways. In cases where plausible departures do not effect meaningful changes in decisions, analysts can reach their decisions with confidence. Otherwise, analysts should have low confidence in the decisions. With this usage in mind, we refer to the framework as a confidence-in-decision (CID) analysis. The CID analysis framework has features in common with the types of sensitivity analyses often done for causal inference ( Fogarty 2023 ) and for imputation of missing data. In causal inference, the goal of a sensitivity analysis typically is to determine how strong the effect of unmeasured confounding must be to nullify an apparent causal relationship ( Rosenbaum 2020 ; VanderWeele & Ding 2017 ; Ding & VanderWeele 2016 ; Schwartz et al. 2012 ; Mathur 2023 ) . With missing data, the goal of a sensitivity analysis typically is to assess how much estimates change under different specifications of missing not at random mechanisms ( Little & Rubin 2019 ; Linero & Daniels 2018 ; Schifeling et al. 2015 ) . Liublinska & Rubin 2014 propose tipping point analysis, which offers a visualization of how the p-value for test of no treatment effect changes under increasingly serious departures from missing at random assumptions. Our work contributes to these approaches by turning the focus explicitly on the impacts of assumptions on ultimate decisions as opposed to statistical estimates alone. The CID analysis framework also is related to statistical decision analysis ( Berger 2013 ) . Decision analyses map statistical estimates to a space of utilities via some function. For example, Suzuki 2022 defines utilities that are a function of the size of the effect of some policy intervention, a probability distribution over possible effect sizes, the cost of implementing the intervention, and the cost of the event that the policy is intended to address. The goal of these analyses is typically to identify some optimal decision or strategy by maximizing the utility function. Our framework aims at revealing the sensitivity of ultimate decisions to the methods chosen to handle data imperfections as opposed to finding an optimal solution. The remainder of this article is organized as follows. In Section 2 , we outline the CID framework. We do in the context of potential measurement error in the input to a predictive model for presidential election outcomes. In Section 3 , we illustrate a a CID analysis with missing data. Here, the analyst considers how strong a missing not at random (MNAR) mechanism must be to change their decision compared to if the data were missing at random (MAR) and handled as such. The context is the percentage of children in North Carolina exposed to high

Read the complete text