Causal Inference for Experiments with Latent Outcomes: Key Results and Their Implications for Design and Analysis

Jiawei Fu, Donald P. Green · 2025

Read Causal Inference for Experiments with Latent Outcomes: Key Results and Their Implications for Design and Analysis by Jiawei Fu, Donald P. Green (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21909.txt.

Introduction

Jiawei Fu and Donald P. Green write for researchers who run trials where the main result cannot be seen at once and must be tracked in more than one flawed way. Grades, surveys, and tests can each hold some error, yet the true result stays hidden.

The authors show a flaw in past methods: the thing being measured can end up not matching across studies. They set out design based steps to pin down the effect of interest, ways to build trials so key claims hold up, and checks on those claims. A worked case shows how more than one measure can sharpen and steady the results.

Catalog record

CREATORJiawei Fu, Donald P. Green
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21909

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Causal Inference for Experiments with Latent Outcomes: Key Results and Their Implications for Design and Analysis Jiawei Fu, Donald P. Green Abstract How should researchers analyze randomized experiments in which the main outcome is measured in multiple ways but each measure contains some degree of error? We describe modeling approaches that enable researchers to identify causal parameters of interest, suggest ways that experimental designs can be augmented so as to make linear latent variable models more credible, and discuss empirical tests of key modeling assumptions. We show that when experimental researchers invest appropriately in multiple outcome measures, an optimally weighted index of the outcome measures enables researchers to obtain efficient and interpretable estimates of causal parameters by applying standard regression methods, and that weights may be obtained using instrumental variables regression. Maximum likelihood and generalized method of moments estimators can be used to obtain estimates and standard errors in a single step. An empirical application illustrates the gains in precision and robustness that multiple outcome measures can provide. 1 Introduction Social scientists often seek to estimate the average causal effect of interventions and increasingly rely on experimental designs to do so convincingly. The statistical literature on experimental design and analysis has grown markedly in recent years, with noteworthy advances in asymptotic theory ( Li & Ding, 2017 ) , modeling ( Sävje et al., 2021 ) , and estimation ( Su et al., 2023 ) . However, one topic of special concern to social scientists has largely escaped attention, even from otherwise comprehensive textbooks: imperfect measurement of experimental outcomes. Angrist & Pischke, 2009 , Gerber & Green, 2012 , and Imbens & Rubin, 2015 offer no sustained formal treatment of outcome measurement or how to analyze experiments in which outcomes are measured in more than one way. The models that these textbooks present implicitly assume that the observed outcome is a manifestation of the true underlying potential outcomes of interest. In the social sciences, however, there is often slippage between the outcomes of interest and the proxy outcome measures at hand. Constructs such as economic inequality, press freedom, political violence, corruption, and post-materialism are just a few examples of latent variables that are thought to be measured imperfectly, despite the sustained efforts of researchers. Our paper builds on recent attempts to formalize the estimation challenges that may arise when latent outcomes are measured with error. Like Stoetzer et al., 2025 , we use potential outcomes notation to unify our discussion of experimental design and outcome measurement. We make four contributions, each of which has important implications for research design and estimation. First, building on a largely forgotten literature dating at least to Costner, 1971 and Alwin & Tessler, 1974 , we show how an intervention’s average treatment effect on a latent outcome can be identified and estimated without “standardizing” the latent outcome in ways that make the units of measurement sample-specific. Adapting ideas from prior work by Bagozzi, 1977 , Sorbom, 1981 , and Kano, 2000 to a potential outcomes framework, we propose efficient and easily implemented estimators for the experimental intervention’s ATE on the latent outcome. Second, we show analytically and through simulation how multiple measures of a latent outcome can improve the precision with which this ATE is estimated. These results highlight an underappreciated practical trade-off between gathering more observations and gathering additional outcome measures. Third, we show how certain data collection strategies can help satisfy what would otherwise be unrealistic measurement assumptions, such as the stipulation that an observed measure represents a linear function of the underlying construct of interest plus measurement error. 1 1 1 Our approach to data collection also reduces reliance on nonlinear models that assume constant treatment effects among all subjects, such as the hierarchical IRT model proposed by Stoetzer et al., 2025 . We offer design recommendations that allow the analyst to stay within a simpler linear modeling framework that accommodates heterogeneous treatment effects but recognize that researchers sometimes have no control over the available outcome measures and may need to take a nonlinear modeling approach. We are by no means the first to note that “invalid” measurements threaten unbiased inference or that redundant outcome measurements can improve precision; our contribution is to bring a coherent analytic framework to the discussion of experimental research design and outcome measurement so as to clarify the value of research designs that allocate resources to the collection of multiple measurements of a latent outcome. Fourth, we show how latent outcome models may be viewed as nested alternatives to seemingly unrelated regression (SUR), allowing researchers to assess empirically whether the constraints imposed by the latent outcome model are consistent with the data. If not, researchers can fall back on the more agnostic SUR approach in which each individual outcome measure is regressed on the treatment. This paper is structured as follows. We begin by presenting a potential outcomes model that defines the target of inference and allows for some degree of slippage between the latent outcome we seek to measure and the observed measures at our disposal. Next, we show formally the conditions under which the average treatment effect on the posited latent outcome is identified. When latent outcomes are measured with random error, this average treatment effect may be estimated consistently, but precision is enhanced when the researcher gathers multiple outcome measures. When measurement errors are systematic – especially when mismeasurement operates differently for subjects assigned to treatment or control – methods that are typically used to analyze experiments may no longer yield unbiased estimates. We show how identification problems may be diagnosed empirically and addressed by diversifying the portfolio of measures and adopting an estimator that leverages the additional measures in a theoretically informed way. A simulated example helps explicate the mechanics of this approach. In order to show its relevance to applied empirical work, we reanalyze the Kalla & Broockman, 2020 experiment, which features a diverse array of outcome measures that improve precision and facilitate robustness checks. 2 Framework Consider a sample of n units drawn from a population of size N . Let Z_{i} denote the treatment assignment for unit i . Generalizing the potential outcomes framework, we assume two potential latent variables: \eta_{i}^{1} is the latent outcome that would be realized for subject i if the treatment were administered and \eta_{i}^{0} is the latent outcome that would be realized for subject i in the absence of treatment. These latent variables are not directly observed; think of them instead as abstract concepts, such as authoritarianism or corruption, that may be measured with some degree of error. Suppose that there are J observed measures of the latent outcome for each subject. Let the j^{th} outcome measure for unit i be Y_{ij}=\lambda_{j}\eta_{i}+\epsilon_{ij}=\lambda_{j}[Z_{i}\eta^{1}_{i}+(1-Z_{i})\eta^{0}_{i}]+\epsilon_{ij} , where \mathbb{E}[\epsilon_{i}]=0 . 2 2 2 This assumption can be relaxed to any constant since outcomes can be mean-centered. Figure 1 illustrates the relationship among the observed and unobserved variables. The observed variables are depicted inside squares, and unobserved variables are depicted inside circles. The latent outcome \eta_{i} can be represented as \eta_{i}=\mathbb{E}\eta_{i}^{0}+\tau Z_{i}+\zeta_{i} , where \zeta_{i} is the idiosyncratic disturbance: \zeta_{i}=\eta_{i}^{0}-\mathbb{E}\eta^{0}_{i}+Z_{i}[(\eta_{i}^{1}-\mathbb{E}\eta^{1}_{i})-(\eta_{i}^{0}-\mathbb{E}\eta^{0}_{i})] . This term has mean zero by construction: \mathbb{E}\zeta_{i}=0 . We are interested in the causal effect of treatment on

Read the complete text