Rerandomization for covariate balance mitigates p p -hacking caused by strategically selecting covariates in regression adjustment
Xin Lu, Peng Ding · 2025
Read Rerandomization for covariate balance mitigates p p -hacking caused by strategically selecting covariates in regression adjustment by Xin Lu, Peng Ding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01137.txt.
Introduction
Xin Lu and Peng Ding make a fresh case for rerandomization in a 2025 statistics paper. Rerandomization means shuffling treatment assignments until the groups look alike on key traits. Its classic selling point, sharper estimates, fades when analysts adjust for traits later anyway, which left the method's theory on shaky ground.
Lu and Ding show a different gain: balance at the design stage guards against p-hacking, where analysts cherry-pick which traits to adjust for until results look significant. False alarms from that cherry-picking shrink under rerandomization. With a strict enough balance threshold, the problem can be solved outright. Their results also guide how tight that threshold should be in practice.
Catalog record
| CREATOR | Xin Lu, Peng Ding |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 201 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rerandomization for covariate balance mitigates p p -hacking caused by strategically selecting covariates in regression adjustment Xin Lu, Peng Ding Abstract Rerandomization enforces covariate balance across treatment groups in the design stage of experiments. Despite its intuitive appeal, its theoretical justification remains unsatisfying because its benefits of improving efficiency for estimating the average treatment effect diminish if we use regression adjustment in the analysis stage. To strengthen the theory of rerandomization, we show that it mitigates false discoveries resulting from p p -hacking caused by strategically selecting covariates in regression adjustment to get more significant p p -values. Moreover, we show that rerandomization with a sufficiently stringent threshold can resolve such p p -hacking. As a byproduct, our theory offers guidance for choosing the threshold in rerandomization in practice. Keywords: average treatment effect; causal inference; experiment design; covariate adjustment; randomized experiment 1 Introduction Randomized experiments are the gold standard for estimating the average treatment effect (ATE). They balance observed and unobserved covariates on average. However, chance covariate imbalances are common in realized treatment allocations, which may complicate the interpretation of the estimated ATE. Rerandomization, termed by Cox (1982) and Morgan and Rubin (2012) , enforces covariate balance in the design stage by rejecting allocations with covariate imbalances, while regression adjustment addresses covariate imbalances in the analysis stage ( Fisher, 1935 ; Freedman, 2008 ; Lin, 2013 ) . Bruhn and McKenzie (2009) conducted a survey of leading experimental researchers in development economics, and suggested that rerandomization is commonly used yet often poorly documented. Gerber and Green (2012) recommended rerandomization as a way to approximate blocking. Although rerandomization is intuitive and has been widely implemented in practice, its necessity has not been fully justified. Since Morgan and Rubin (2012) , most papers on rerandomization have focused on the aspect of efficiency for estimating the ATE. That is, rerandomization decreases the variances of the estimators for the ATE, if the covariates are predictive of the outcomes ( Branson et al., 2016 ; Morgan and Rubin, 2012 ; Li et al., 2018 ; Wang and Li, 2022 ; Wang et al., 2023 ; Li and Ding, 2020 ) . However, those papers demonstrated that when we use the same set of covariates in rerandomization and regression adjustment, rerandomization cannot further improve efficiency compared with solely regression adjustment. Given regression adjustment, the additional benefits of rerandomization remain unclear if the criterion is the efficiency for estimating the ATE. We depart from the current literature on efficiency and offer a new perspective. In particular, we demonstrate that rerandomization can mitigate p -hacking caused by strategically selecting covariates in regression adjustment. The term p -hacking, coined by Simmons et al. (2011) , refers to the practices that a researcher might use to generate more significant p -values ( Brodeur et al., 2020 ) . It has become a pervasive and profound issue, undermining the credibility and reproducibility of scientific discoveries ( John et al., 2012 ; Ioannidis, 2005 ) . Among the practices of p -hacking, strategically selecting covariates is especially noteworthy, as it is common for researchers to explore various covariate combinations in the analysis stage ( Simmons et al., 2011 ) . While a detailed pre-analysis plan can prevent such a problem, it can be costly and therefore is not required for every study ( Banerjee et al., 2020 ) . In this paper, we focus exclusively on p -hacking caused by strategically selecting covariates in regression adjustment. Throughout this paper, the term p -hacking refers exclusively to this specific practice. Our contributions are twofold. First, we study two rerandomization schemes: rerandomization with Mahalanobis distance (ReM) ( Cox, 1982 ; Morgan and Rubin, 2012 ) and rerandomization based on the p -values of marginal t -tests (ReP) ( Gerber and Green, 2012 ; Zhao and Ding, 2024 ) . We demonstrate that rerandomization can lower the type I error rate due to p -hacking, and as rerandomization enforces nearly perfect balancing of the covariates, it can control the type I error rate. The intuition is that when we balance the covariates by rerandomization, the p -values from regression adjustments with different covariate combinations tend to be coherent with each other; see Zhao and Ding (2024) for more discussion about rerandomization and coherence. Therefore, p -hacking will not distort the p -values too much under rerandomization. Second, our theory gives recommendations for the rerandomization thresholds to control the type I error rates. To emphasize, although rerandomization can provide a safeguard against p -hacking caused by strategically selecting covariates, we do not encourage researchers to base their inference on a hacked p -value. Researchers should prespecify their primary analysis whenever possible. Notation: Let \mathbb{P}_{\infty} denote the probability measure with respect to the asymptotic distribution. For two finite-population arrays \{\boldsymbol{a}_{i}\}_{i=1}^{n} and \{\boldsymbol{b}_{i}\}_{i=1}^{n} , define the finite-population mean \bar{\boldsymbol{a}}=n^{-1}\sum_{i=1}^{n}\boldsymbol{a}_{i} and covariance \boldsymbol{S}_{\boldsymbol{a}\boldsymbol{b}}=(n-1)^{-1}\sum_{i=1}^{n}(\boldsymbol{a}_{i}-\bar{\boldsymbol{a}})(\boldsymbol{b}_{i}-\bar{\boldsymbol{b}})^{\top}. Let \Phi(\cdot) be the cumulative distribution function of a standard Gaussian distribution. Let z_{1-\alpha} be the \alpha th upper quantile of a standard Gaussian distribution. Let \chi^{2}_{K} be the \chi^{2} distribution with degrees of freedom K . Let \chi_{K,1-\alpha}^{2} be the upper \alpha th quantile of \chi^{2}_{K} . For a set of tuples \{(u_{i},\boldsymbol{V}_{i1},\ldots,\boldsymbol{V}_{iL}):u_{i}\in\mathbb{R},\boldsymbol{V}_{il}\in\mathbb{R}^{K_{l}},i=1,\ldots,n,l=1,\ldots,L\} , let \operatorname{lm}(u_{i}\sim\boldsymbol{V}_{i1}+\cdots+\boldsymbol{V}_{iL}) be the ordinary least squares (OLS) fit of u_{i} on (\boldsymbol{V}_{i1},\ldots,\boldsymbol{V}_{iL}) . 2 Design-based framework for hacked p -values Consider a completely randomized experiment (CRE) with n_{z} units, z=0,1 , assigned to the treatment group z , and n_{1}+n_{0}=n . Let Y_{i}(z) be the potential outcome of unit i under the treatment group z and Z_{i}\in\{0,1\} be the treatment indicator of unit i . The observed outcome is Y_{i}=Y_{i}(1)Z_{i}+Y_{i}(0)(1-Z_{i}) . Let \tau_{i}=Y_{i}(1)-Y_{i}(0) be the individual level treatment effect. For unit i , we observe K covariates, \boldsymbol{x}_{i}=({x}_{i1},\ldots,{x}_{iK}) . We are interested in estimating the ATE: \bar{\tau}=n^{-1}\sum_{i=1}^{n}\tau_{i}. We focus on the design-based inference framework, which relies solely on the randomness of treatment assignment, conditional on the potential outcomes and covariates ( Neyman, 1990 ; Freedman, 2008 ; Imbens and Rubin, 2015 ) . Consequently, conclusions drawn from this framework do not depend on the underlying stochastic process of the potential outcomes and covariates; see Li and Ding (2017) for a review. Fisher (1935) considered testing the sharp null hypothesis: H_{0\textsc{f}}:Y_{i}(1)=Y_{i}(0),\quad i=1,\ldots,n. Although we observe only one of the two potential outcomes for each unit, the sharp null hypothesis enables us to impute all the missing potential outcomes and implement the Fisher randomization test. Specifically, for any test statistic, we can compute its finite-sample exact p -value by randomly permuting the treatment indicators to simulate the randomization distribution. Neyman (1990) considered the weak null hypothesis: H_{0\textsc{n}}:\bar{\tau}=0. The common choices of the test statistics for the weak null hypothesis are the studentized ATE estimators. For example, the coefficient estimator of Z_{i} from the following OLS regression is an unbiased estimator of \bar{\tau} ( Freedman, 2008 , Theorem 1) : \displaystyle\operatorname{lm}(Y_{i}\sim 1+Z_{i}), (1) which we denote by \hat{\tau}=n_{1}^{-1}\sum_{i:Z_{i}=1}Y_{i}-n_{0}^{-1}\sum_{i:Z_{i}=0}Y_{i} . The corresponding t -statistic using the associated Eicker–Huber–White (EHW) standard error is asymptotically valid for testing H_{0\textsc{n}} ( Ding, 2024a , chap. 4) . With covariates \boldsymbol{x}_{i} , regression adjustment is often used in the analysis stage to reduce the variance of the ATE estimator, therefore improving the power of the t -test (