Optimal Sample Splitting for Observational Studies
Qishuo Yin, Dylan S. Small · 2026
Read Optimal Sample Splitting for Observational Studies by Qishuo Yin, Dylan S. Small (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22782.txt.
Introduction
Qishuo Yin and Dylan S. Small look at how the design of an observational study shapes its trust. In such studies, a treatment was not assigned by chance, so hidden traits may bias the results. The authors ask how a study should split its sample to test that bias well.
A split sample lets one part build the study plan and the other part test it. How the split is done changes how much hidden bias the study can stand before its findings fall apart. Good splitting rules give health and social research a fairer test of cause and effect.
Catalog record
| CREATOR | Qishuo Yin, Dylan S. Small |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22782
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimal Sample Splitting for Observational Studies Qishuo Yin, Dylan S. Small Abstract In observational studies of treatment effects, estimates may be biased by unmeasured confounders, which can potentially affect the validity of the results. Understanding sensitivity to such biases helps assess how unmeasured confounding impacts credibility. The design of an observational study strongly influences its sensitivity to bias. Previous work has shown that the sensitivity to bias can be reduced by dividing a dataset into a planning sample and a larger analysis sample, where the planning sample guides design decisions. But the choice of what fraction of the data to put in the planning sample vs. the analysis sample was ad hoc. Here, we develop an approach to find the optimal fraction using plasmode datasets. We show that our method works well in high-dimensional outcome spaces. We apply our method to study the effects of exposure to second-hand smoke in children. The OptimalSampling R package implementing our method is available at GitHub . Keywords: Causal Inference, Multiple Hypothesis Tests, Sample Split, Second-hand Smoke, Sensitivity Analysis, Treatment Effect. 1 Introduction Causal inference is easiest in a randomized experiment because each unit has a known probability of assignment to treatment or control, and this provides a basis for inference ( 8 ) . However, randomized experiments are sometimes infeasible or unethical, and observational studies must be considered. In observational studies, the treatment probabilities are unknown, and there may be unmeasured confounders ( 6 ) . Matching methods, inspired by stratified randomization in randomized experiments, enable randomization inference methods to be used if there are no unmeasured confounders. However, there may be unmeasured confounders which can cause treatment effect estimates to be biased. Sensitivity analysis can be used to measure how sensitive findings (that assume no unmeasured confounding) are to bias from unmeasured confounders. The sensitivity of an observational study to bias can be substantially affected by its design ( 32 ) . For example, if there are multiple outcomes, choosing certain of the outcomes to focus on can substantially decrease sensitivity to bias as discussed in 15 . The paper introduced a sample splitting approach in which a pilot sample is used to choose the design, and then the remaining part of the sample is used for analysis. They showed that the approach can substantially reduce sensitivity to bias. However, its performance can vary significantly depending on the chosen sample split fraction. In this paper, we propose a method that determines the optimal sample splitting fraction which is adaptive to both low and high dimensional outcomes space. Our method numerically optimizes the sample splitting fraction for a semi-synthetic data set. We also provide implementation guidance for the practitioner with empirically recommended splitting fractions that have close to optimal performance across a range of settings. We apply our method to assess the health effects of children exposed to second-hand smoke. Such an assessment must rely on observational data due to ethical constraints preventing randomized trials ( 36 ; 13 ; 39 ) . 21 analyzed the relationship between second-hand smoke exposure and blood lead levels using the Third National Health and Nutrition Examination Survey. More recent work 4 suggests that exposure to second-hand smoke influences multiple biological pathways beyond blood lead levels. Because this evidence is from observational studies and there may be unmeasured confounders, a sensitivity analysis is useful. Following previous studies ( 21 ) , 46 of the health effects of children’s exposure to second-hand smoke, we used matching to control for aspects of a child’s demographics and socioeconomic status (gender, race/ethnicity, age, education of the reference adult in the child’s household, poverty level of the child’s family, number of rooms in the house and family size). However, there may still be unmeasured covariates that we may fail to take into consideration when conducting matching. Then, each child in a matched pair may not live in exactly the same environment as the other in the same pair. To address this, we performed a sensitivity analysis to examine how children’s health is influenced by second-hand smoke under different bias factors in the assignment of treatment. This sensitivity analysis will give a conclusion about which outcomes are affected by second-hand smoke even allowing for some unmeasured confounding. Section 5 presents the detailed results. Sensitivity analysis plays a central role in assessing the robustness of causal conclusions in observational studies. Various sensitivity analysis methods have been developed for different contexts ( 24 ; 25 ; 26 ; 27 ; 14 ; 43 ; 42 ; 22 ; 45 ; 48 ) . We develop our method for Rosenbaum’s sensitivity analysis framework, which is particularly useful for matched observational studies. Our approach provides a unified framework that provide the solution of optimal sample splitting fractions, accommodates high-dimensional outcome spaces through multiple testing procedures, and improves design sensitivity in observational studies. This paper proceeds as follows. Section 2 introduces the notation and sensitivity model. Section 3 develops the optimal sample split method for multiple outcome tests. Section 4 presents the simulation results in different settings. Finally, Section 5 applies our methodology to study the health effects of second-hand smoke on children. 2 Notation and Review 2.1 Paired Randomized Experiments Consider I matched pairs of two individuals, indexed by j=1,2 within each pair i=1,\dots,I . We analyze K outcomes of interest, indexed by k=1,\dots,K . Let \mathbf{x}_{ij} denote the observed pretreatment covariates, u_{ij} the unobserved covariates, and Z_{ij} the observed treatment assignment. Define the observed response vector for the k^{\text{th}} outcome as \mathbf{R}_{k}=\left(R_{11k},R_{12k},\dots,R_{I1k},R_{I2k}\right)^{T} . Treatment assignment is binary: Z_{ij}=1 indicates treatment, and Z_{ij}=0 indicates control. Within each matched pair, exactly one individual receives treatment and the other receives control, ensuring Z_{i1}+Z_{i2}=1 for all i . Matching is conducted on observed covariates, i.e., \mathbf{x}_{i1}=\mathbf{x}_{i2} , though unobserved confounders may differ, i.e., u_{i1}\neq u_{i2} . For each individual in pair i , the potential outcomes are denoted as r_{T_{ik}} (under treatment) and r_{C_{ik}} (under control). The k^{\text{th}} observed response is therefore: \displaystyle R_{ijk}=Z_{ij}r_{T_{ik}}+(1-Z_{ij})r_{C_{ik}},\quad k=1,\dots,K. Define \mathbf{r}_{C_{k}}=(r_{C_{1k}},\dots,r_{C_{Ik}})^{T} and \mathbf{r}_{T_{k}}=(r_{T_{1k}},\dots,r_{T_{Ik}})^{T} . The treatment effect for pair i on outcome k is given by: \displaystyle\delta_{ik}=r_{T_{ik}}-r_{C_{ik}}. Fisher’s sharp null hypothesis ( 8 ) for outcome k asserts that the treatment has no effect on outcome k for all subjects: \displaystyle H_{0}^{k}:\delta_{ik}=0,\quad\forall i\quad\text{ for outcome }k. Define the set of all possible treatment assignments as: \displaystyle\mathcal{Z}=\{\mathbf{Z}=(Z_{11},Z_{12},\dots,Z_{I2})\mid Z_{ij}\in\{0,1\},\forall i,j\}, which contains |\mathcal{Z}|=2^{I} elements. Let the sigma-algebra generated by all variables in the dataset be \mathcal{F}=\sigma\{(\boldsymbol{x}_{ij},u_{ij},r_{T_{k}},r_{C_{k}},\,k=1,\cdots,K),\,i=1,\cdots,I,\,j=1,2\} . In a paired randomized experiment, each unit’s treatment assignment is independent and equally probable: \displaystyle\mathrm{Pr}(Z_{i1}=1\mid\mathcal{F})=\frac{1}{2},\quad Z_{i2}=1-Z_{i1}. Thus, the joint probability of a specific assignment is: \displaystyle\mathrm{Pr}(\mathbf{Z}=\mathbf{z}\mid\mathcal{F},\mathcal{Z})=\frac{1}{|\mathcal{Z}|}=\frac{1}{2^{I}}. Under Fisher’s sharp null hypothesis, a test statistic T_{k} for outcome k is given by T_{k}=t(\mathbf{Z},\mathbf{R}_{k}) . The significance level of the test follows: \displaystyle\mathrm{Pr}\left(T_{k}\geq c\mid\mathcal{F},\mathcal{Z}\right)=\frac{\left|\{\mathbf{z}\in\mathcal{Z}:\,t(\mathbf{Z},\mathbf{R}_{k})\geq c\}\right|}{2^{I}}. (1) 2.2 Sensitivity Analysis In observational studies, treatment assignment may be influenced by unobserved confounders ( 29 ) . This results in an assignment probability \pi_{i}=\mathrm{Pr}(Z_{i1}\mid\mathcal{F}) that deviates from \frac{1}{2} , i.e., \pi_{i}\neq\frac{1}{2} . Sensitivity analysis assesses how varying levels of unmeasured confounding impact inferences about treatment effects. Two commonly employed approaches are sensitivity models in treatment assignments and sensitivity models in unobserved covariates. The sensitivity model in treatment assignments is quantified by the odds ratio