Causal Inference in Counterbalanced Within-Subjects Designs
Justin Ho, Harvard University · 2025
Read Causal Inference in Counterbalanced Within-Subjects Designs by Justin Ho, Harvard University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03937.txt.
Introduction
Justin Ho and Jonathan Min examine a beloved experimental design and show the assumption it quietly leans on. Their 2025 paper concerns within-subjects studies where every participant receives both treatment and control in sequence, with the order shuffled across people to cancel carryover effects. That cancellation works only if carryover is symmetric, an assumption rarely checked.
Using the potential outcomes framework, the authors name the extra condition honest inference needs: a form of sequential exchangeability between the periods. They then offer a toolkit for living with the problem. Diagnostic checks can probe the assumption, washout periods and covariate adjustment can soften violations, and different designs can replace counterbalancing where it cannot be trusted. The paper's message is not that the design is useless, but that its convenience has a price that should be stated and, where possible, paid down deliberately.
Catalog record
| CREATOR | Justin Ho, Harvard University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03937
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Causal Inference in Counterbalanced Within-Subjects Designs Justin Ho, Harvard University Abstract Experimental designs are fundamental for estimating causal effects. In some fields, within-subjects designs, which expose participants to both control and treatment at different time periods, are used to address practical and logistical concerns. Counterbalancing, a common technique in within-subjects designs, aims to remove carryover effects by randomizing treatment sequences. Despite its appeal, counterbalancing relies on the assumption that carryover effects are symmetric and cancel out, which is often unverifiable a priori . In this paper, we formalize the challenges of counterbalanced within-subjects designs using the potential outcomes framework. We introduce sequential exchangeability as an additional identification assumption necessary for valid causal inference in these designs. To address identification concerns, we propose diagnostic checks, the use of washout periods, and covariate adjustments, and alternative experimental designs to counterbalanced within-subjects design. Our findings demonstrate the limitations of counterbalancing and provide guidance on when and how within-subjects designs can be appropriately used for causal inference. Keywords: Carryover effects, Sequential exchangeability, Potential outcomes framework, Treatment effect identification, Within-subject experimental design, Causal inference assumptions 1 Introduction ”Failure to conclude that a model is false must be a failure of our imagination, not a success of the model.” — McElreath (2018) Experimental designs are increasingly used in the social and behavioral sciences in estimating the causal effects of various interventions. The randomized controlled trial (RCT) is widely regarded to be the ‘gold standard’ when it comes to estimating the causal effects of treatment as opposed to estimates from an observational study. Typically, participants are divided into distinct treatment and control groups, allowing for straightforward comparisons of intervention outcomes in RCTs. 1 1 1 This design is called between-subjects design in some fields since interventions are randomly assigned between subjects ( Maxwell et al., 2017 ) . However, this conventional approach is not the only experimental design used. Within-subjects design, also known as repeated-measures design, offers an alternative that exposes all participants to every condition in a study. This design is sometimes used in fields such as experimental psychology ( Keren, 2014 ) , management sciences ( Lane et al., 2024 ) , political science ( Clifford et al., 2021 ) , and even medical trials ( Sarkies et al., 2019 ) , which enables researchers to compare responses across treatments for the same individuals. Experimenters often employ these methods when they face logistical challenges in recruitment, implementation (particularly regarding the costs associated with specific trials), or when the population concerned is relatively small, and the experimenter wishes to ensure there is common support in terms of the exposure of the treatment with respect to the covariates ( Maxwell et al., 2017 ; Erlebacher, 1977 ). However, this experimental design introduces challenges in causal inference. For example, imagine a study testing the effect of background music on concentration. Participants complete two problem-solving tasks: one in silence and one with music. If every participant always does the silent condition first and the music condition second, any observed difference in performance might not be due to the music itself but rather to other factors. For instance, they might perform better in the second task simply because they have become more familiar with the problem format (a practice effect) or worse due to mental fatigue. Since these order-related influences are confounded with the treatment, the researcher cannot tell whether any observed changes are truly caused by the music or just by the passage of time and repeated testing. The chief culprit confounding the estimate of the treatment effect in within-subjects designs are carryover effects , which are the residual effects from the first period that influence responses in the second period ( Maxwell et al., 2017 ) . To address these concerns, counterbalancing is often applied to within-subjects designs. In clinical trials, this design is also known as crossover trials or AB/BA designs ( Zhang et al., 2022 ; Sibbald and Roberts, 1998 ; Matthews, 1994 ) . This approach randomizes the order of treatment and control conditions for participants, assuming that this would cancel out any carryover effects, resulting in an unbiased estimate. Figure 1 shows how a counterbalanced within-subjects design is implemented in the case where there are two experimental conditions. However, this assumption is often overly optimistic. Counterbalancing assumes that carryover effects are symmetric and cancel out, yet this symmetry is unverifiable a priori . Differential carryover effects, where one condition’s impact persists or interacts with subsequent conditions in an asymmetric manner, can bias estimates of the average treatment effect even when participants are properly counterbalanced. There are some advantages in using within-subjects design in some empirical settings, as it aligns more naturally with economic and psychological theories that model how individuals respond to changing conditions. For example, in empirical studies testing utility theories and preference, participants might exposed to multiple conditions of the study, allowing the experimenter to observe how an individual makes such a tradeoff ( Charness et al., 2012 ) . Despite their inferential limitations, counterbalanced within-subjects designs offer several practical advantages. They allow researchers to collect multiple observations per participant, improving statistical efficiency—especially useful when working with small populations or when recruitment is costly or difficult, such as in studies involving venture capitalists ( Lane et al., 2024 ) . In such cases, a between-subjects design may suffer from issues like limited overlap or lack of common support across covariates, making within-subjects designs an attractive alternative despite their potential identification problems. In this paper, we analyze the implications of using counterbalanced within-subjects designs for causal inference, analyzing the problem with the potential outcomes framework. Specifically, we first explore the assumptions required in counterbalanced within-subjects design to produce treatment effect estimates consistent with those obtained from between-subjects design in the case of two discrete time periods, with the goal of estimating the Average Treatment Effect (ATE) as defined in the potential outcomes framework. We then introduce sequential exchangeability as a generalized assumption required, alongside the standard assumptions of the Rubin Causal Model, to identify the causal effects of treatment in other sequential designs similar to a counterbalanced within-subjects design. We introduce sequential randomization and selective sequential randomization in Section 3 as alternative sequential designs where the assumptions of sequential exchangeability is more credible than the commonly used counterbalanced within-subjects design. Sequential designs are often used to improve the efficiency and reliability of experiments. For example, Whitehead et al. (2020) use a design that allows dropping underperforming treatments mid-trial, Zhou et al. (2018) propose a rerandomization method to improve covariate balance over time, and Tamura et al. (2011) apply a sequential parallel design to reduce placebo effects. While these approaches differ in motivation, they all rely on using data collected across stages to better estimate treatment effects. We focus on how similar ideas apply to counterbalanced within-subjects designs and relate sequential exchangeability as a condition under which such designs can identify the average treatment effect (ATE). We want to emphasize that such assumptions are typically difficult to verify a priori and hence, we do not advocate the use of such experimental designs to estimate ATE, even in the case of sequential randomization where the assumptions are more credible. However, in situations where such a