Reevaluating Causal Estimation Methods with Data from a Product Release
Justin Young, Muthoni Ngatia, Eleanor W. Dillon · 2026
Read Reevaluating Causal Estimation Methods with Data from a Product Release by Justin Young, Muthoni Ngatia, Eleanor W. Dillon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11845.txt.
Introduction
Justin Young, Muthoni Ngatia, and Eleanor W. Dillon test how well causal tools work when the truth is known. New machine learning tools make it easier to model links between traits, treatments, and outcomes. That makes causal claims from plain data look more safe.
They use a rare data set with two parts. One part is a true trial roll-out of a new product trait at a large tech firm. The other part is users who chose the trait on their own. By matching tool calls to the trial truth, they can grade each tool fairly.
Catalog record
| CREATOR | Justin Young, Muthoni Ngatia, Eleanor W. Dillon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11845
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reevaluating Causal Estimation Methods with Data from a Product Release Thanks: For helpful comments, we are grateful to Guido Imbens, Jann Spiess, Vasilis Syrgkanis, Lihua Lei, Julian Nyarko, Stefan Wager, Ramesh Johari, Jason Weitze, Lea Bottmer, David Ritzwoller, Fabio Vera, and conference and seminar participants at NABE TEC 2024, Microsoft Research, and Stanford. Justin Young, Muthoni Ngatia, Eleanor W. Dillon Abstract Recent developments in causal machine learning methods have made it easier to estimate flexible relationships between confounders, treatments and outcomes, making unconfoundedness assumptions in causal analysis more palatable. How successful are these approaches in recovering ground truth baselines? In this paper we analyze a new data sample including an experimental rollout of a new feature at a large technology company and a simultaneous sample of users who endogenously opted into the feature. We find that recovering ground truth causal effects is feasible—but only with careful modeling choices. Our results build on the observational causal literature beginning with 29 , offering best practices for more credible treatment effect estimation in modern, high-dimensional datasets. 1 Introduction Many econometric techniques aim to estimate causal effects from existing, nonrandomized data. These methods are an important tool to answer applied problems in economics, social science, and business where randomized experiments are infeasible or unethical. Unlike prediction problems, where estimates can eventually be compared with realized outcomes, causal effect estimates typically have no ground truth to compare against, leaving open questions about their practical reliability. Evaluation of these methods against a small set of existing benchmarks, starting with LaLonde’s seminal 1986 examination of a job training program, have produced mixed results. Our study evaluates observational causal inference methods using a new benchmark dataset from Microsoft. In this dataset, a new product feature (masked to allow the data to be released) was rolled out via a randomized experiment for a subset of users, while simultaneously being made available to other users who could choose whether to adopt it. We measure the same set of device performance outcomes in both samples, along with a rich set of user and device characteristics. This creates an ideal testing ground: we observe both an experimental “gold standard” estimate and an analogous observational sample where uptake is nonrandom. Our large and homogenous data samples improve on many of the earlier benchmarks that had limited covariate overlap between treated and untreated units in the observational sample and/or limited similarity between the observational and experimental samples. We use the richness of the data to explore the performance of a range of causal estimation techniques, including modern methods that leverage flexible machine learning, estimating both the average treatment effects (ATE) and conditional average treatment effects (CATE). Our goal is to provide practical advice for researchers: analyzing which causal methods perform well on nonrandomized data, how these methods can be used most effectively, and what checks and tests provide meaningful signals about the quality of estimated effects. We estimate the causal effect of the new product feature on two outcomes related to device performance. In both cases, the simple difference in mean outcomes across users in the observational sample who did and did not adopt the new features tells the same qualitative story as the experimental benchmark: the new feature causes a small increase in the continuous performance outcome and a larger decrease in the binary performance outcome. 1 1 1 The details of the performance outcomes are also masked at the request of Microsoft to enable the data to be released. Positive changes do not necessarily align with “good” outcomes. However, in both cases these naive mean differences miss the experimentally-estimated causal effects by a meaningful margin, indicating substantial endogenous selection. For the continuous outcome, we are able to remove all the selection bias and recover the experimental benchmark using the available device and user characteristics and carefully deployed causal estimation methods. For the binary outcome, none of our attempts come close to the experimental benchmark. Our analysis yields several bits of advice for practitioners. First, observational causal estimation methods can recover true causal effects when used carefully, but the details of estimation matter. When we follow established best practices, we find that doubly robust estimators that use flexible machine learning to capture the effects of confounders on treatment and outcome outperform simpler methods like linear regression. However, this flexibility can become a liability when those best practices are not followed. Most machine learning models have hyperparameters that shape the estimation to prevent overfitting in finite samples. 2 2 2 Examples include the free parameters that determine the degree of regularization in LASSO regression or the maximum depth of trees in a random forest model. These hyperparameters should be tuned to each data sample using sample splitting to measure out-of-sample performance. When we skip this preliminary step, doubly robust methods can deliver estimated causal effects from the observational sample that are as biased as the unconditional means–indicating that choosing the right estimation model is as important as measuring the right set of confounders. To our knowledge, this is the first real-world application that quantifies the importance of estimating the effects of confounders on treatments and outcomes precisely, supplementing the simulated illustrations in 6 ; 8 . We also reinforce the importance of trimming the observational sample to remove “always treated" or “never treated" types ( 13 ; 12 ; 24 ) and averaging across estimates to reduce model uncertainty ( 7 ; 35 ) . Fortunately, each of these best practices describe concrete, verifiable steps that can be executed with confidence in applied settings, even those without ground truth benchmarks. The methods we use to estimate causal treatment effects from nonrandomized data all rely on an unconfoundedness assumption ( 37 ) : that we have measured enough covariates to control for any dependence between treatment assignment and the realization of outcomes. Our second guidance for practitioners is to use multiple metrics to assess the plausibility of this core assumption, which cannot be tested directly. The new product release we study has a very strong effect on the binary performance metric. In the observational sample, we overstate the true treatment effect even using our best methods, estimating declines in the probability of the outcome of 54% rather than the true 43%, indicating that we may be missing some key confounders. We conduct sensitivity analysis following 9 , commonly used as one check of unconfoundedness. Our estimated treatment effects on the binary outcome strongly “pass" the most common test: they are unlikely to be pushed all the way to zero by an unobserved confounder because the effect is too strong. When only the existence and direction of the effect matters, this check may provide sufficient reassurance. However, when magnitudes matter, for example when treatment is costly, we recommend additional checks including considering the fit of the preliminary prediction models (our available confounders explain none of the variation in the binary outcome) and additional sensitivity checks against benchmarks other than zero. Finally, we use the scale and richness of our data to conduct a novel real-world test of conditional average treatment effect estimators. We validate the DR-score ( 10 ) method to detect heterogeneous treatment effects. In both the experimental and observational samples, this test suggests meaningful