Evaluating Prediction-based Interventions

with Human Decision Makers In Mind · 2025

Read Evaluating Prediction-based Interventions by with Human Decision Makers In Mind (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05704.txt.

Introduction

Inioluwa Deborah Raji and Lydia Liu examine how experiments on prediction tools can mislead when people stay in the loop. Their 2025 paper notes that automated decision systems now advise judges, doctors, and other decision makers in weighty settings. Tests of these tools often assume people use predictions in a simple, steady way.

The authors build formal models of several plausible ways people respond to a model aid. Each model links decisions across subjects, breaking the independence that standard trial math relies on. That breakage can shift measured effect sizes and bias the verdict on whether the tool helps. The work aims to make evaluation designs honest about human behavior, so deployments rest on sounder proof.

Catalog record

CREATORwith Human Decision Makers In Mind
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05704

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Prediction-based Interventions with Human Decision Makers In Mind Abstract Automated decision systems (ADS) are broadly deployed to inform and support human decision-making across a wide range of consequential settings. However, various context-specific details complicate the goal of establishing meaningful experimental evaluations for prediction-based interventions. Notably, current experiment designs rely on simplifying assumptions about human decision making in order to derive causal estimates. In reality, specific experimental design decisions m may induce cognitive biases in human decision makers, which could then significantly alter the observed effect sizes of the prediction intervention. In this paper, we formalize and investigate various models of human decision-making in the presence of a predictive model aid. We show that each of these behavioral models produces dependencies across decision subjects and results in the violation of existing assumptions, with consequences for treatment effect estimation. This work aims to further advance the scientific validity of intervention-based evaluation schemes for the assessment of ADS deployments. Inioluwa Deborah Raji Lydia Liu University of California, Berkeley Princeton University 1 INTRODUCTION When making use of a predictive model in a decision-making process, human discretion often influences the final decisions – hence, there is "not necessarily a one-to-one mapping from the predictive tool to the final outcome” ( Albright 2019 ) . A growing body of research hints at why – while the field of machine learning is focused on the evaluation of predictive model outputs using accuracy metrics often tested on benchmark datasets ( Raji et al. 2021 ) , these assessments do not reveal the full pattern of how humans interact with the algorithm ( Green and Chen 2019c ) , and thus provides little assurance that the predictions will improve downstream outcomes on the impacted population through better decision-making ( Liu et al. 2023 ; Liu et al. 2024 ) . Kleinberg et al. 2018b goes further to illustrate how, unlike prediction evaluation which focuses on minimizing model generalization error, decisions involve a more complex calculation of cost-benefit trade-offs, requiring considerations beyond just model performance. This difference between predictions and decisions - also sometimes referred to as judgements ( Agrawal et al. 2018 ) - necessitates a critical expansion of scope in terms of what is being evaluated. Once deployed, predictive models operate much more like typical policy interventions, not isolated prediction problems. These prediction-based interventions are thus ideally evaluated around the causal question of how much the introduction of the predictive system impacts some important downstream outcome. This paradigm is about hypothesis testing a treatment - effectively thinking through the counterfactual measurement of what happens in the presence and in absence of the prediction-based intervention for a given decision-maker on a specific case. What we actually need to evaluate is not just the accuracy of the models’ output predictions relative to outcomes, but the impact of such predictions on the actions of decision-makers and the appropriateness of those decisions. One obvious approach to the estimation of these causal effects is through the implementation of experimental evaluations for prediction-based interventions. Several efforts across criminal justice Imai et al. 2023 , healthcare ( Plana et al. 2022a ) , and education ( Dawson et al. 2017a ) have begun conducting such evaluations to estimate the impact of introducing a prediction-based intervention to modify a given default policy. However, it is clear that those conducting these experiments not necessarily factoring in the particularities of the prediction-based intervention context into their experiment design. We argue that increased attentiveness to experiment design choices in this context is important – as such choices may impact the responsiveness (i.e., tendency to be swayed by an algorithmic decision aid) of the decision-makers relying on algorithmic recommendations, which in turn distorts our understanding of the average treatment effect of prediction-based interventions. Our contributions are thus as follows: 1. We examine the role of specific experiment design choices on judge responsiveness – specifically, choices that experiment designers make on (1) treatment assignment ( Z ), (2) the model positive prediction rate ( P(\hat{Y}=1) ) (by setting the threshold of the model corresponding to a recommended action) and (3) the model correctness ( P(\hat{Y}=Y) ) (through model selection). We conclude that beyond intrinsic judge biases, experiment design choices can also impact judge responsiveness . 2. We mathematically formalize how such experimental design choices impact judge responsiveness via novel models of judge decision making and prove that this leads to violations of SUTVA. We furthermore discuss how these design choices and their impact on judge responsiveness can lead to a mis-estimation of the average treatment effect in experimental evaluation settings for prediction-based interventions. 3. Using existing experiment data of a prediction-based intervention ( Imai et al. 2020 ) , we simulate a scenario of modifying experimental design choices and observing differences in judge decisions and estimated average treatment effect. 2 RELATED WORK Here we provide a high level overview of background literature – further discussion on related work is available in Appendix E . Modeling Discretion in Quasi-experimental investigations. It is well known that decision-makers are subject to intuitive biases that inform how they respond to cases more generally ( Guthrie et al. 2007 ; Guthrie et al. 2000 ) , and specifically how they factor in information from algorithmic decision aids ( DeMichele et al. 2021 ) . However, most experimental and quasi-experimental work that have considered judge responsiveness assume that the tendency to follow or not follow the algorithmic recommendation is inherent to each judge – a static attribute of the judges’ internal state, or their access to privileged information, i.e. factors outside of the experimenter’s control ( Mullainathan and Obermeyer 2022 ; Albright 2019 ) . For example, Hoffman et al. 2018b considered possible access to privileged information that impacted the hiring manager’s judgement (each hiring manager having a particular exception rate for making a different decision from the algorithm). In contrast, we consider a dynamic model of judge responsiveness that is sensitive to factors in experimental design, acknowledging the connection between these experiment design factors and the induced behavioral biases that may impact judge decision-making. The Rise of Experimental Evaluations. In this work, we focus on Imai et al. 2020 , who presented a case study involving a Randomized control trial (RCT) examining the use of Public Safety Assessment (PSA) risk scores by judges in Dane County, Wisconsin. However, various other examples of RCTs for risk assessments have been explored in other domains. For instance, Wilder et al. 2021b ran a “clinical trial” on an algorithm used to select peer leaders in a public health program from homelessness youth. Similarly, several studies in education execute RCTs to assess models used to predict which students (in real-world and MOOC settings) are most likely to drop out ( Dawson et al. 2017b ; Borrella et al. 2019b ) . In healthcare, Plana et al. 2022b provides a recent survey of the 41 RCTs of AI/ML-based healthcare interventions done to date, critiquing a systematic lack of adherence to documentation standards such as CONSORT-AI and SPIRIT-AI ( Liu et al. 2020 ) . That being said, many of the RCTs cited suffer from the same design flaws and challenges we observe in Imai et al. 2020 , which we argue

Read the complete text