Assessing model error in counterfactual worlds Journal: TBD
Emily Howerton, Justin Lessler · 2025
Read Assessing model error in counterfactual worlds Journal: TBD by Emily Howerton, Justin Lessler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00836.txt.
Introduction
Emily Howerton and Justin Lessler wrote a 2025 paper on a common planning habit: asking what would happen if. Counterfactual scenarios help leaders plan for outbreaks, policies, and other futures. Yet the projections these exercises produce are rarely checked later against what really happened.
When a projection misses, the error can come from two places. The real world may drift from the assumed scenario, or the model itself may be miscalibrated. Howerton and Lessler work on telling those two apart, which they call assessing model error in counterfactual worlds. The split matters, because each kind of error calls for a different fix.
Catalog record
| CREATOR | Emily Howerton, Justin Lessler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-00836
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assessing model error in counterfactual worlds Journal: TBD Emily Howerton, Justin Lessler Abstract Counterfactual scenario modeling exercises that ask “what would happen if?" are one of the most common ways we plan for the future. Despite their ubiquity in planning and decision making, scenario projections are rarely evaluated retrospectively. Differences between projections and observations come from two sources: scenario deviation and model miscalibration. We argue the latter is most important for assessing the value of models in decision making, but requires estimating model error in counterfactual worlds. Here we present and contrast three approaches for estimating this error, and demonstrate the benefits and limitations of each in a simulation experiment. We provide recommendations for the estimation of counterfactual error and discuss the components of scenario design that are required to make scenario projections evaluable. 1 Introduction Predictive models are a central tool in many fields for assessing the state of the world and planning for the future. Models enable us to not only forecast what will happen but also to predict what would happen if certain conditions were to occur. For instance, what if we limit global surface warming to 2°C? Or, what if 80% of seniors received a vaccination against seasonal influenza? By helping us to determine our best possible answer to these questions given current knowledge, models play an essential role in managing risk and informing decisions. These what would happen if scenario projections are most useful when we are interested in the effects of decisions or key uncertainties [ 7 ] . When we aim to intervene in the present to change outcomes in the future, scenario projections can help quantify the benefits and costs of potential actions. If key drivers of dynamics are highly uncertain, scenario projections can help define future outcomes under different assumptions (e.g., best and worst cases). In many situations we are making decisions under uncertainty, hence are simultaneously interested in both. Although predictions of the future are typically generated to inform planning and response in the present, retrospective evaluation of predictions is essential to improving predictive models [ 1 ] and building trust with end users [ 6 ] . There is well-developed theory and methodology for evaluating forecasts [ 2 ] , where forecasts of what will happen are compared to observations that eventually materialize. The goal of this assessment is to evaluate whether forecasts are “well calibrated", or consistent with observations. These evaluation metrics, however, cannot be straightforwardly applied to scenario projections. When we make projections under multiple scenarios, at most one set of scenario assumptions will be true, and likely none will hold exactly. We cannot directly evaluate the calibration of projections for these counterfactual worlds that we never observe. For example, an airline might project annual earnings under scenarios of a 2% and 5% increase in travel activity, but in actuality see a 3% increase for which they had no projection. Because of this fundamental challenge, scenario projections are rarely evaluated in practice. However, without evaluation, biases in the models can go unidentified and mislead decision making. Climate modelers have long grappled with this challenge. Because the scenarios that were assumed cannot be directly evaluated against observed data, they have instead proposed a range of other metrics for quality assessment and trust building. These include, for example, the ability of a model to reliably recreate past conditions or its consistency with physical laws [ 8 ] . More recently, a few attempts to directly evaluate scenario projections have been made in climate [ 3 ] and for infectious diseases [ 5 ] . However, as we will discuss below, these methods each have limitations. Here, we present a conceptual framework for evaluating scenario projections. We describe the central challenge with scenario projection evaluation, situate existing methods within our conceptual framework, and propose three general approaches to overcome these challenges, each with their own requirements and assumptions. Finally, we demonstrate the feasibility and limitations of each approach in a simulation experiment. 2 The challenge with evaluating scenario projections: Two sources of error The fundamental challenge with evaluating the calibration of scenario projections arises from their counterfactual nature. Because a scenario projection specifies what would happen if a particular set of conditions were to be realized, the explicit comparison of a projection to an observation (as in forecast evaluation) is difficult to interpret because there are two potential sources contributing to the error: scenario deviation and model miscalibration. To illustrate this fundamental challenge, we present a conceptual schematic of a scenario projection ( Figure 1 ). Let \mathcal{P}^{m}(y|x_{i}) be a projection of outcome y from model m under conditions x_{i} specified along scenario axis x for scenario i . For example, consider a case where low and high vaccination coverage scenarios are of interest (x-axis, Figure 1 ), and a model generates projections of cumulative hospitalizations for each of these assumptions over a 6-month period (black points, Figure 1 ). After the projection period has passed, we observe the realized vaccination coverage, x^{*} , and the corresponding realized cumulative hospitalizations, y^{*}=\mathcal{P}^{*}(y|x^{*}) (red point, Figure 1 ). In a forecasting setting, x^{*} is irrelevant because the forecast of what will happen is expected to encompass all possible future values of x (and, implicitly, their likelihood). However, for a scenario projection, which is conditional on x_{i} , x^{*} will match at best only one of the scenarios that were modeled, and usually x^{*} will not match any of the scenarios modeled. Despite this mismatch, it might be tempting to compare the projection \mathcal{P}^{m}(y|x_{i}) and the observation y^{*} directly. For example, if scenario projections and observations are plotted together, this is the natural comparison. Importantly, though, this comparison is difficult to interpret because there are two potential drivers of differences between the projected values and observation ( Figure 1 ): 1. scenario deviation: the world did not unfold according to the scenarios modeled (i.e., x_{i}\neq x^{*} ). Thus, projected values from any scenario, \mathcal{P}^{m}(y|x_{i}) , should not necessarily match observed values from the “realized" scenario, \mathcal{P}^{*}(y|x^{*}) . 2. model miscalibration: even if the projections had been made for the perfect scenario (i.e., x_{i}=x^{*} , white circle Figure 1 ), the projection may still deviate from observations. That is, model miscalibration is the error we would observe in the setting of no scenario deviation . Because decisions will be based on projections in the modeled counterfactual scenarios specified, we argue that model miscalibration is the most relevant source of error and the best measure of whether underlying models are performing as intended. That is, we care most about the difference between the projected value and the value that would have been observed in the scenarios that were modeled, i.e., e^{m}(x_{i})=\mathcal{P}^{m}(y|x_{i})-\mathcal{P}^{*}(y|x_{i}) . To make these comparisons, though, half of the necessary information will always be missing. For example, information is missing even if we aim to calculate error in the scenario that was realized, e^{m}(x^{*})=\mathcal{P}^{m}(y|x^{*})-\mathcal{P}^{*}(y|x^{*}) , the projected value \mathcal{P}^{m}(y|x^{*}) is likely not available in the original set of projections. It may be possible to retrospectively obtain this value by rerunning the models at the realized scenario, x^{*} , or by estimating the projected value for the scenario (e.g., through a surrogate model of the