Statistical Decision Theory with Counterfactual Loss

Benedikt Koch, Kosuke Imai · 2025

Read Statistical Decision Theory with Counterfactual Loss by Benedikt Koch, Kosuke Imai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08908.txt.

Introduction

Benedikt Koch and Kosuke Imai extended numeric choice theory in 2025 to judge choices against the road not taken. Classic theory grades a treatment policy by the results it made, ignoring what would have happened under the alternatives, yet some choices. Like a judge's choice to release or jail a defendant before trial, are only meaningful unit by unit, against that person's feasible alternatives. Counterfactual risk captures that idea, but it is usually impossible to identify from data, since each unit reveals only one result.

The authors prove a sharp identification result: under strong ignorability, counterfactual risk can be identified exactly when the loss function is additive across potential results. With more than two treatments on the table, additive counterfactual losses can recommend varied actions than standard losses. And they encode not just accuracy but how hard each choice was, information classic theory throws away.

Catalog record

CREATORBenedikt Koch, Kosuke Imai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE125 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-08908

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Statistical Decision Theory with Counterfactual Loss Thanks: We thank Iav Bojinov, Peter Buisseret, Nathan Cheng, Neil Christy, Johann Gaebler, Anvit Garg, D. James Greiner, Amanda Kowalski, Wenqi Shi, Sooahn Shin, Tomasz Strzalecki, Davide Viviano, Théo Voldoire, and an anonymous reviewer of IQSS’s rapidPeer for insightful discussions. Benedikt Koch, Kosuke Imai Abstract Many researchers apply classical statistical decision theory to evaluate treatment choices and learn optimal policies. However, because this framework relies solely on realized outcomes under chosen actions and ignores counterfactuals, it cannot assess the quality of a decision relative to feasible alternatives at the unit level, which is an important requirement in some settings. For example, in pretrial bail decisions, a judge must balance crime prevention upon release against the risk of imposing unnecessary burdens on arrestees. A central challenge in this framework is identification: since only one potential outcome is observed per unit, counterfactual risk is typically not identifiable. We show that, under strong ignorability, counterfactual risk is identifiable if and only if the loss is additive in the potential outcomes. We further demonstrate that additive counterfactual losses can yield treatment recommendations that differ from those based on standard losses when more than two treatment options are available. We show that additive counterfactual losses capture not only decision accuracy but also decision difficulty, whereas standard losses reflect accuracy alone. Finally, we introduce a symbolic linear inverse program that determines whether a given counterfactual loss yields an identifiable risk, without requiring data. Keywords: causal inference, decision theory, counterfactual utility, identification, policy learning, potential outcomes, risk 1 Introduction Statistical decision theory, pioneered by Wald [1950] , formulates decision-making as a game against nature. A decision-maker selects a decision D=d in the face of an unknown state of nature \theta , and incurs a prespecified loss (or negative utility) \ell(d;\theta) , which quantifies the consequence of the chosen decision. A better decision corresponds to a lower loss. Given covariates \bm{X} , we can define a decision rule D=\pi(\bm{X}) that maps observed data to decisions. The performance of a decision rule is commonly evaluated by its risk , that is, the expected loss: R(\pi;\theta,\ell)\ :=\ \mathbb{E}\left[\ell(\pi(\bm{X});\theta)\right], where the expectation is taken over the distribution of \bm{X} . Since \theta is unknown, the true risk is typically unidentifiable. This motivates the use of optimality criteria such as admissibility, minimax, minimax-regret, and Bayes risk [ Wald, 1950 , Savage, 1972 , Berger, 1985 ] . Statistical decision theory also provides a foundational framework for classical estimation theory [ Lehmann and Casella, 2006 , e.g.,] and hypothesis testing [ Lehmann et al., 1986 , e.g.,] . In a series of influential papers, Manski [2000] , Manski [2004] , Manski [2011] applies this framework to the problem of treatment choice. In its simplest form, the goal is to choose a treatment D to minimize the expected loss associated with an outcome of interest Y . Because the treatment may influence the outcome, the loss \ell(d;Y(d)) depends on the potential outcome Y(d) under the chosen treatment d\in\mathcal{D} and possibly the treatment itself, where \mathcal{D} denotes the support of decision D [ Neyman, 1923 , Rubin, 1974 ] . As before, we can define a treatment rule D=\pi(\bm{X}) based on observed covariates, and evaluate its performance via the following risk: R(\pi;\ell)\ =\ \mathbb{E}{\left[\ell(\pi(\bm{X});Y(\pi(\bm{X})))\right]}, (1) where the expectation is taken over the joint distribution of \bm{X} and Y(\pi(\bm{X})) . Since only one potential outcome is observed per unit, i.e., Y=Y(D) , the risk is generally unidentifiable unless the treatment rule coincides with the one used to generate the data. Consequently, additional assumptions, such as the strong ignorability of treatment assignment [ Rosenbaum and Rubin, 1983 ] , are required to identify the risk and learn optimal treatment rules [ Manski, 2000 , Kitagawa and Tetenov, 2018 , Athey and Wager, 2021 , e.g.,] . In this paper, we generalize the standard framework of statistical decision theory for treatment choice by introducing a broader class of loss functions. Specifically, we consider counterfactual losses, which depend on the full set of potential outcomes rather than the outcome under the selected treatment alone. That is, the loss takes the form of \ell(d;\{Y(d^{\prime})\}_{d^{\prime}\in\mathcal{D}}) . We refer to this as a counterfactual loss because it evaluates each decision in light of the alternative outcomes that could have occurred had a different treatment been chosen. Given this generalized loss, we define the counterfactual risk for measuring the performance of treatment rule \pi as: R(\pi;\ell)\ =\ \mathbb{E}[\ell(\pi(\bm{X});\{Y(d)\}_{d\in\mathcal{D}})], (2) where the expectation is taken over the distribution of covariates \bm{X} and all potential outcomes \{Y(d)\}_{d\in\mathcal{D}} . Counterfactual losses raise two foundational questions: coherence and identification. Coherence asks whether decision making based on counterfactual losses satisfies basic rationality requirements such as transitivity. In a companion paper [ Koch et al., 2026 ] , we establish an axiomatic foundation for counterfactual loss by showing that the preferences it induces are coherent and satisfy a generalization of the von Neumann-–Morgenstern axioms [ von Neumann and Morgenstern, 1944 ] . We briefly summarize this result in Section 4 . In this paper, we take the coherence of counterfactual loss as given and study its identification. In contrast to the standard risk in Equation ( 1 ), even under strong ignorability, the counterfactual risk in Equation ( 2 ) is generally not identifiable, due to its dependence on the joint distribution of potential outcomes. We derive a structural condition on the counterfactual loss that enables the identification of counterfactual risk (Section 3 ). Specifically, we consider a class of counterfactual losses that are additive in the potential outcomes. We show that under strong ignorability of treatment assignment, additivity is both necessary and sufficient for identifying counterfactual risk differences , and hence for identifying an optimal policy within a given policy class. Importantly, our results extend beyond the binary treatment and outcome cases that have been a focus of the existing literature. We next examine how restrictive the additivity condition is (Section 5 ). When decisions are binary, i.e., |\mathcal{D}|=2 , we show that every additive counterfactual loss admits an equivalent standard loss that induces the same ranking over policies and hence the same optimal decision rule. By contrast, when the decision problem involves more than two treatment options ( |\mathcal{D}|>2 ), additive counterfactual losses can induce rankings and optimal policies that cannot be replicated by any standard loss. This divergence arises because additive counterfactual losses capture both the accuracy and the difficulty of a decision, whereas standard losses reflect accuracy alone. Furthermore, we show that, under binary outcomes and a monotonicity restriction, every counterfactual loss is additive. Finally, we formulate a symbolic linear inverse problem that offers a practical test for whether a given loss is additive and hence its risk identifiable (Section 6 ). This test does not require any data, and the solution to this linear inverse problem provides a closed-form decomposition of counterfactual risk into identifiable marginal distributions of potential outcomes. Related Literature. The application of counterfactual logic to decision-theoretic problems has appeared in several strands of the literature. One of the most prominent examples of counterfactual loss is the notion of regret, \ell(d;\{Y(d^{\prime})\}_{d^{\prime}\in\mathcal{D}})=\max_{d^{\prime}\in\mathcal{D}}Y(d^{\prime})-Y(d) [ Savage, 1972 ] . Yet, because of the aforementioned identification challenge, prior work has

Read the complete text