Leveraging Large Language Models to Improve Precision in Randomized Controlled Trials
Jaylin Lowe, Adam Sales, Johann A. Gagnon-Bartsch · 2026
Read Leveraging Large Language Models to Improve Precision in Randomized Controlled Trials by Jaylin Lowe, Adam Sales, Johann A. Gagnon-Bartsch (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30157.txt.
Introduction
Jaylin Lowe, Adam Sales and Johann A. Gagnon-Bartsch test whether language-model predictions can sharpen randomized controlled trials without breaking rigour. Trial precision often suffers when good predictors are missing, yet model predictions are known for bias and error. Their pipeline folds LLM predictions into trial analysis following the observational-data approach, and they apply it to three case studies. The predictions safely improve precision, especially when trials lack good predictors or carry text data that suit language models.
Catalog record
| CREATOR | Jaylin Lowe, Adam Sales, Johann A. Gagnon-Bartsch |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42637 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Large Language Models to Improve Precision in Randomized Controlled Trials Jaylin Lowe, Adam Sales, Johann A. Gagnon-Bartsch Abstract Large language models (LLMs) are increasingly used in statistical research and applications. However,they are also notorious for unreliable or biased information. Here, we explore whether LLMs can be used to improve the precision of randomized controlled trials (RCTs) in a safe and rigorous way. Following similar work on leveraging observational data, we incorporate LLM predictions into an RCT analysis. While incorporating external predictions to improve precision is not new, the value of using LLM predictions in this manner is an open question. We develop a pipeline for best leveraging LLM predictions in this context and apply it to three different case studies. We find that these predictions can safely improve precision, particularly when the RCT lacks predictive covariates or contains covariates—such as text data—that are well-suited to LLMs. Introduction Large language models (LLMs) are now widely used for a variety of tasks [ 7 , 9 ] . Despite these accomplishments, many researchers have expressed concerns that LLMs hallucinate, respond inconsistently to prompts, and amplify training data biases [ majeed_reliability_2024 , 8 , 5 , 10 ] . In this paper, we investigate whether LLMs can be used to improve precision in randomized controlled trials (RCTs) in a safe and rigorous way. We focus on how to do this in practice. RCTs have long been considered the “gold standard” in causal inference. However, RCTs are often small, especially in the social sciences, leading to estimates with high variance [ evans_how_2022 , kraft_interpreting_2020 ] . This issue is often addressed via covariate adjustment, discussed more in the next section. Covariates are measured characteristics recorded for each observation that may be associated with the outcome; covariate adjustment models small chance imbalances in these characteristics across groups [ 3 ] . The problem with only using RCT data for covariate adjustment is that the covariates and outcomes may be hard to model. For instance, the covariates could be high-dimensional, have a complex structure, or be in a form, like text, that is not compatible with standard statistical models. In those cases, the RCT sample size often is not large enough on its own to estimate a predictive model. To address these issues, other researchers have developed methods for improving precision in RCTs by integrating RCT data with larger external datasets [ 1 , gagnon-bartsch_precise_2023-1 , gui_combining_2024 , 4 , rosenman_combining_2023 , rosenman_propensity_2022 ] . Large external datasets can be used to produce predicted outcomes, which can be included in the model as an additional univariate covariate. Ideally, these predicted outcomes will be highly predictive of the outcome. In essence, we use large external datasets to fit complex models that distill these difficult covariates into a small set of conventional covariates suitable for covariate adjustment estimators. However, external datasets are not always available. In those cases, we could potentially use an LLM in place of an external dataset. This may be especially helpful when we are dealing with covariates well-suited to LLMs, such as text covariates. We will explore the feasibility of using LLMs instead of external data in the approach outlined in [ gagnon-bartsch_precise_2023-1 ] . This approach is guaranteed to produce an unbiased estimator and will not harm precision in moderate to large samples. Thus, we need not rely on LLM accuracy to ensure unbiasedness, nor does LLM inaccuracy compromise precision. In principle, using an LLM generated prediction as a covariate in the RCT analysis is fairly straightforward. However, determining the best methods to construct these covariates is an open question. In this paper, we present a pipeline for generating LLM predictions and apply it to three different case studies: a natural experiment comparing recidivism rates between judges, an RCT evaluating a computer-based curriculum for Algebra I, and an RCT estimating the effect of open-access on citation counts for scientific papers. We find that in the first two examples, which feature rectangular, quantitative covariate data, our method does little to improve precision, but in the last example, where we use abstracts as covariates, our method substantially improves precision. In one case, the improvement is equivalent to increasing the sample size by nearly 60%. This paper is organized as follows. First, we introduce the relevant causal inference notation and provide an overview of alternative methods for improving precision in RCTs. Next, we introduce the three case studies. Our Methods section gives our approach for obtaining useful LLM predictions and evaluating their utility. Then, we apply the method to each of the studies, display the results, and discuss the impact of different case studies on the results. In our Limitations section, we discuss a potential threat to the method’s validity and provide evidence that it is not a major concern. The final section, Discussion, concludes. Background Randomized Controlled Trials In a randomized experiment, our goal is to estimate the effect of a treatment on a particular outcome. We will use the potential outcomes framework, developed by Neyman and Rubin [ rubin_causal_2005 , splawa-neyman_application_1990 ] . Under this framework, we have N observations, indexed from i=1,....N . Each observation is randomly assigned to treatment ( Z_{i}=1 ) or control ( Z_{i}=0 ), where Z is the vector of treatment assignments. We assume a Bernoulli experiment, where all units are independently randomized with P(Z_{i}=1)=p . We have a vector of RCT covariates \mathbf{x}_{i} for each observation. Let n_{t} and n_{c} represent the number of observations assigned to treatment and control, respectively. Each observation has a potential outcome under treatment, y_{i}^{t} , and a potential outcome under control, y_{i}^{c} . The potential outcomes y_{i}^{t} and y_{i}^{c} represent the outcome value we would observe if observation i were assigned to treatment or control, respectively. The individual treatment effect \tau_{i} is defined as y_{i}^{t}-y_{i}^{c} . We want to estimate the average treatment effect: \displaystyle\bar{\tau}=\frac{1}{N}\sum_{i=1}^{N}\tau_{i}=\frac{1}{N}\sum_{i=1}^{N}(y_{i}^{t}-y_{i}^{c}) (1) We only observe y_{i}^{t} or y_{i}^{c} for each observation, but never both. Thus, \bar{\tau} is not directly calculable, but may be estimated. Consider the Horvitz-Thompson estimator [ horvitz_generalization_1952 , aronow_class_2013 ] , which estimates the average treatment effect as: \displaystyle\hat{\tau}=\frac{1}{N}\sum_{i=1}^{N}Z_{i}\cdot\frac{Y_{i}}{p}-\frac{1}{N}\sum_{i=1}^{N}(1-Z_{i})\cdot\frac{Y_{i}}{1-p} (2) That is, we estimate the average treatment effect as the difference between the average of the outcome in the treatment group and the average of the outcome in the control group, weighted by the assignment probability. In the next section, we discuss alternatives to this estimator that help reduce the variance of our estimate. Improving Precision in RCTs Random assignment to the treatment and control groups ensures that all covariates will be balanced between the groups in expectation. However, covariates can still be imbalanced between the groups by chance, especially in smaller samples, leading to high variance. Covariate adjustment addresses this imbalance by leveraging the RCT covariates to account for chance imbalances, leading to a more precise estimator. One simple option for covariate adjustment is to run a linear model regressing the outcome on the treatment indicator and the covariates [ 3 , freedman_regression_2008 ] . If the covariates are predictive of the outcome, doing so can improve the precision of the estimators. Others have proposed improvements to this general framework [ lin_agnostic_2013 ] . The most relevant example is model imputation estimators, which