A Weighting Framework for Clusters as Confounders in Observational Studies Eli Ben-Michael, Avi Feller, Luke Keele Abstract When units in observational studies are clustered in groups, such as students in schools or patients in hospitals, researchers often address confounding by adjusting for cluster-level covariates or cluster membership. In this paper, we develop a unified weighting framework that clarifies how different estimation methods control two distinct sources of imbalance: global balance (differences between treated and control units across clusters) and local balance (differences within clusters). We show that inverse propensity score weighting (IPW) with a random effects propensity score model — the current standard in the literature — targets only global balance and constant level shifts across clusters, but imposes no constraints on local balance. We then present two approaches that target both forms of balance. First, hierarchical balancing weights directly control global and local balance through a constrained optimization problem. Second, building on the recently proposed Generalized Mundlak approach, we develop a novel Mundlak balancing weights estimator that adjusts for cluster-level sufficient statistics rather than cluster indicators; this approach can accommodate small clusters where all units are treated or untreated. Critically, these approaches rest on different assumptions: hierarchical balancing weights require only that treatment is ignorable given covariates and cluster membership, while Mundlak methods additionally require an exponential family structure. We then compare these methods in a simulation study and in two applications in education and health services research that exhibit very different cluster structures. Keywords: Partial Pooling, Weighting, Fixed Effects, Random Effects, Causal Inference 1 Introduction In standard observational causal inference, researchers assume that there are no unmeasured confounders after adjusting for individual-level covariates alone. In many observational studies, however, individuals are clustered in groups, such as students nested in schools or patients nested in hospitals. In such cases, researchers can weaken this assumption by additionally adjusting for cluster-level covariates or for cluster membership itself. Consider an observational study evaluating a new surgical procedure, as we do in Section 8 . In addition to adjusting for patient-level covariates such as age, measures of health, and comorbidities, we might worry that hospital-level factors also influence both the treatment decision and the outcome. For example, the new procedure may be used more often in hospitals in which surgeons receive more advanced training. To capture this, we might want to additionally adjust for a hospital-level measure of surgical training. Alternatively, we might want to adjust for the admitting hospital itself, which could also capture other important aspects of “unmeasured context” ( 3 ) . This simple example highlights some of the many trade-offs researchers face when adjusting for confounding in this setting. This paper reviews the many weighting estimators that have been developed to account for cluster-level confounding in observational causal inference. Building on several reviews of model-based inverse propensity score weighting (IPW) in this clustered setting ( 36 ; 13 ; 17 ) , we focus on adapting two new methodological developments to this area. First, we consider balancing weights , a popular alternative to model-based IP weights that instead uses constrained optimization to directly find weights that balance covariates between groups ( 8 ; 7 ) . We show that extant balancing weights and model-based IPW methods primarily target a combination of global balance —covariate differences between treated and control units across clusters—and local balance —covariate differences between treated and control units within clusters. Model-based IPW with a random cluster intercept, by far the most common approach in the literature, implicitly targets global balance alone and does not include any control of local balance. By contrast, both model-based IPW with random coefficients ( 3 ) and hierarchical balancing weights ( 7 ) target both global and local balance. A major challenge is that achieving both excellent global and local balance is often infeasible due to data sparsity; local balance within some clusters may be especially difficult to achieve due to limited cluster sample sizes. We show that several methods require dropping small clusters or clusters with nearly all treated or control units, which shifts the target estimand. Second, we consider Generalized Mundlak Estimation ( 2 ) , which adjusts for cluster-level sufficient statistics in place of adjusting for cluster membership itself. The key assumption is that treatment assignment is ignorable given individual-level information and cluster-level sufficient statistics (Assumption 3 ) . This is implied by ignorable treatment assignment given individual-level information and cluster membership, along with an additional assumption that the joint distribution of individual-level covariates and treatments follows an exponential family distribution. Unlike methods that adjust for cluster membership, the Mundlak approach can retain smaller clusters, albeit at the cost of imposing the stronger exponential family assumption that requires the cluster-level sufficient statistics to fully capture the relevant cluster-level information. We then propose Mundlak balancing weights , a balancing weights estimator that also adjusts for cluster-level sufficient statistics. We outline two variants of this estimator, which constrain global balance in different ways, and argue that these are attractive estimators in practice. Finally, we illustrate the trade-offs between these estimators with a simulation study and with two empirical illustrations with different cluster structures. The first application is an observational study of the impact of smaller class sizes on test scores, in which students are nested in schools with relatively small cluster sizes. The second application is an observational study of emergency general surgery for hepato-pancreato-biliary disorders, in which patients are nested within hospitals with many large clusters. We show that trade-offs differ in these two settings, and offer some suggestions for practice. Our article is organized as follows. In Section 2 we outline notation, estimands, and identification assumptions, and review common weighting estimation approaches for causal inference. In Section 3 , we develop our unified framework for global and local balance. In Section 4 , we use this framework to analyze extant methods that condition on cluster membership. In Section 5 , we use the framework to analyze methods that instead condition on cluster-level covariates. In Section 6 , we outline inference and bias correction. In Section 7 we conduct a simulation study. In Section 8 , we analyze two different empirical applications. Method Balance Target Main Limitation Methods that adjust for cluster membership (Assumption 2) Cluster-Specific Intercepts Global only No control over local balance Stratified Analysis Local only Must drop clusters with no treatment variation Random Coefficients Global + Local Computationally difficult with convergence issues Hierarchical balancing weights Global + Local Must drop clusters with all/no treated units Methods that adjust for cluster-level covariates (Assumption 3; stronger than Assumption 2) Generalized Mundlak Estimator Global + Local Relies on model-based propensity score; Requires exponential family assumption and specification of sufficient statistics Mundlak Balancing Weights Global + Local Requires exponential family assumption and specification of sufficient statistics Table 1: Overview of Weighting Methods for Cluster Adjustment 2 Preliminaries 2.1 Notation and problem setup We first outline our notation and problem setup, most of which is standard for describing an observational study with individualized treatment assignment. We assume we have a sample of units i=1,\dots,n , drawn i.i.d. from some population, and denote Z_{i} as a binary treatment where Z_{i}=1 corresponds to the treated condition, and Z_{i}=0 corresponds to the control condition; n_{1} denotes the number of units assigned to treatment, n_{0} denotes the number assigned to control. We let Y_{i}\in\mathbb{R} denote the observed outcome. We use X_{i}\in\mathcal{X} to denote a vector of baseline covariates. To reduce notational complexity, we will drop the i subscript when taking expectations. Next, we define some additional notation specific to the context where it is thought that group or cluster membership is related to treatment assignment; we use the terms group or cluster interchangeably to denote how units are nested. Let the group indicator G_{i}\in\{1,\ldots,K\} denote membership for unit i in that group. This notation might describe, for example, students in schools or patients in hospitals. We use n_{1g} and n_{0g} to denote the number of treated and control units in group G_{i}=g . Denote the propensity score , the probability of receiving treatment conditional on the baseline covariates, as e(x)\equiv P(Z=1\mid X=x) . And define the conditional expected outcome m(z,x)\equiv\mathbb{E}[Y\mid Z=z,X=x] . We will also consider the propensity score and conditional expected potential outcome when additionally conditioning on group membership: e(x,g)\equiv P(Z=1\mid X=x,G=g) and m(z,x,g)\equiv\mathbb{E}[Y\mid Z=z,X=x,G=g] . 2.2 Assumptions and estimands We outline the key set of assumptions for an observational study, and how the presence of clusters changes the identification of treatment effects. First, we make the standard Stable Unit Treatment Value Assumption ( 43 ) that there are no hidden forms of treatment and that a subject’s potential outcome is not affected by other subjects’ treatment. We therefore rule out applications with clustered interference, where units interact within groups ( 27 , see, for example) . We can then define each unit’s two potential responses, (Y_{i}(1),Y_{i}(0))\in\mathbb{R}^{2} , corresponding to their response under treatment and control. These assumptions imply that the observed outcome is Y_{i}=Z_{i}Y_{i}(1)+(1-Z_{i})Y_{i}(0) . Our focus will be on expected contrasts between potential outcomes or counterfactual states. In observational studies the target of inference is often the Average Treatment effect on the Treated (ATT), \tau\equiv\mathbb{E}[Y(1)-Y(0)\mid Z=1]=\mu_{1}-\mu_{0} , where we denote \mu_{1}\equiv\mathbb{E}[Y(1)\mid Z=1] and \mu_{0}\equiv\mathbb{E}[Y(0)\mid Z=0] . Estimating \mu_{1} is straightforward: the average observed outcome among the treated units is an unbiased estimate. The difficulty then lies in estimating \mu_{0} , corresponding to the expected counterfactual outcome under control for the treated units. We focus on the ATT for brevity, but our results generalize to other linear estimands, including the Average Treatment Effect (ATE) and the Average Treatment effect on the Overlap population ( 35 , ATO;) . Ignorability without cluster membership. In observational studies targeting the ATT without cluster-level information, the standard assumption is strong ignorability ( 41 ) . This combines (i) ignorable treatment assignment —treatment is independent of the potential outcomes given the individual-level covariates—and (ii) overlap —no individual deterministically receives treatment. Assumption 1 (Strong Ignorability) . Y(1),Y(0)\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z|X and e(x)<1 for all x\in\mathcal{X} Note that other estimands have different overlap requirements, e.g. for the ATE we would require a stronger assumption that 00 . When \rho_{U}=0 , there is no cluster-level confounding and methods that account for clusters are unnecessary. When \rho_{U}>0 , cluster-based methods should outperform methods that ignore cluster membership. We compare four methods: • Balancing weights with no cluster adjustment. A baseline that does not adjust for clusters. • Model-based IPW with random intercept (RI-IPW). The standard approach in the literature. As discussed in Section 4.1 , this targets global balance only. Incorporating random coefficients was infeasible in practice. • Hierarchical balancing weights. The estimator from 6 , described in Section 4.2 , which targets both global and local balance by conditioning on cluster membership. • Mundlak balancing weights. The estimator from Section 5.2 , which targets cluster-level sufficient statistics rather than cluster membership. We implemented balancing weights using the balancer library in R ( 5 ) and measured the performance of each method via: (1) standardized absolute bias, \left|\mathbb{E}[\widehat{\text{effect}}-\text{effect}]/\mathbb{E}[\text{effect}]\right| ; and (2) the root mean-squared error (RMSE), \sqrt{\mathbb{E}[(\widehat{\text{effect}}-\text{effect})^{2}]} . We repeated the simulation 1,000 times. Figure 1 : Bias and Root Mean Square Error of balancing weights, IPW based on a random intercept model, hierarchical balancing weights, and Mundlak weights, varying the degree of cluster confounding. Figure 1 presents the results. When there is no cluster-level confounding ( \rho_{U}=0 ), hierarchical balancing weights shows some bias and has the highest RMSE; RI-IPW is also biased and has the next highest RMSE. Thus, there is some cost to accounting for clusters when clusters are irrelevant. By contrast, standard balancing weights and Mundlak weights are both unbiased in this scenario—unsurprising for standard balancing weights, but notable for Mundlak weights. Once we introduce cluster-level confounding, bias for standard balancing weights grows substantially, as expected. More surprising is that bias for RI-IPW also increases and actually exceeds the bias for standard balancing weights; this pattern persists at the highest level of confounding. At moderate confounding, hierarchical balancing weights and Mundlak weights perform nearly identically. At the highest level of confounding, bias for hierarchical balancing weights increases, while Mundlak weights remain nearly unbiased. These results yield three insights. First, RI-IPW performs poorly across all conditions. While some bias when clusters are irrelevant might be expected, the high bias when cluster-level confounding is present is striking: RI-IPW fails to outperform a method that ignores clusters entirely. Focusing on global balance alone does little to reduce bias from cluster-level confounding. Second, hierarchical balancing weights effectively account for cluster-level confounders but incur a cost when clusters are irrelevant. Third, Mundlak weights perform well across all scenarios: they remain nearly unbiased whether or not cluster-level confounding is present, avoiding the penalty that other cluster-based methods pay when clusters are irrelevant. 8 Applications We now present results from two applications with contrasting cluster structures: an education study examining pre-school programs and test scores, and a health services study examining emergency general surgery. In both cases, units are nested within clusters and the goal is to condition on clusters to capture unmeasured context. The applications differ in cluster size: the education application has many clusters but few units per cluster, while the health application has many clusters with widely varying sizes, including some quite large clusters. For both applications, we compare four methods that adjust for observed confounders, as described in Section 7 : (1) balancing weights with no cluster adjustment; (2) model-based IPW with a random intercept; (3) hierarchical balancing weights; and (4) Mundlak balancing weights. These comparisons allow us to assess the effect of cluster adjustment relative to individual-level balancing alone, and the added value of local balance constraints beyond RI-IPW, which only enforces global balance and is widely used in applied work. We also use agreement between hierarchical and Mundlak balancing weights as an informal check on the plausibility of the stronger Assumption 3 : since Mundlak methods require this additional exponential family structure while hierarchical methods do not, substantial disagreement between these estimates could indicate that the chosen sufficient statistics do not fully capture cluster-level confounding. We report select results in the main text. For the Mundlak methods, we implement two versions: Global balance (GB) , which enforces global balance on cluster-level covariates, and Average-to-one (AvTO) , which constrains weights to average to one within each cluster. Since we found limited differences between these two approaches, we report GB estimates in the main text and defer AvTO results to the supplementary materials. We also estimated treatment effects after bias-correction with outcome models, which made minor differences; these results also appear in the supplementary materials. Next, we review how we specify the basis functions. First, \phi(x) is comprised of the unit level covariates, with splines fits added to those continuous covariates with nonlinear associations with the outcome. Next, for the Mundlak weights, the basis functions are more complex. Here, we condition on \phi(x) , z\phi(x) , and \psi(s,x) , where \phi(x) is the set of unit level covariates, z\phi(x) is set of the first order interactions between \phi(x) and the proportion treated and control in each cluster, and \psi(s,x) is the set of first order interactions between \phi(x) and \bar{S}_{G_{i}} , which is vector the group means of the unit level covariates. We use the balancer library in R ( 5 ) for all methods except model-based IP weights. We selected all hyperparameter values using a data driven method based on a regression of Y on X restricted to the control group (Z=0) . The hyperparameter is the variance of the residuals from this regression ( 5 ) . For each application, we estimate the ATT and associated 95% confidence interval. We also report the effective sample size (ESS) for \hat{\mu}_{0} (since we are targeting the ATT): methods that balance more features tend to have lower ESS and thus higher variance. Balance metrics. We report several global and local balance metrics. First, we use a common metric known as the standardized mean difference for covariate j , \text{SMD}_{j} , and the cluster-specific version, \text{SMD}_{g,j} : \text{SMD}_{j}=\frac{\bar{\phi}^{(1)}_{j}-\bar{\phi}^{(0)}_{j}}{s_{j}}\qquad\text{SMD}_{g,j}=\frac{\bar{\phi}^{(1)}_{g,j}-\bar{\phi}^{(0)}_{g,j}}{s_{g,j}} where \bar{\phi}^{(1)}_{j} is the mean in the treated group; \bar{\phi}^{(0)}_{j} is the weighted mean in the control group; s_{j} is the pooled standard deviation, s_{j}=\sqrt{\frac{1}{2}\left[\left(s^{(1)}_{j}\right)^{2}+\left(s^{(0)}_{j}\right)^{2}\right]} ; and g denotes the cluster-specific versions. We also compute an L_{2} measure of imbalance across covariates both globally and locally: L_{2}^{\text{global}}=\sqrt{\frac{1}{p}\sum_{j=1}^{p}\left(\text{SMD}_{j}\right)^{2}},\qquad\qquad L_{2}^{\text{local}}=\sqrt{\frac{1}{PK}\sum_{j=1}^{p}\sum_{g=1}^{K}\text{SMD}_{g,j}^{2}} We can compare how each method reduces the amount of imbalance by calculating the percent reduction in the L_{2} metric with and without weighting: we call this the percentage bias reduction (PBR). We should note that we calculate multiple L_{2} values, since different methods balance different features of the covariates and as such are not always directly comparable. 8.1 Education: Small Clusters The first application focuses on the impact of pre-school programs on post-kindergarten math test scores from 15 and 34 . Here, students are nested in schools, and we wish to account for school context in the estimation of the treatment effects. That is, the analytic goal is to use schools to account for aspects of “unmeasured context” that affect test scores: school membership should capture many parts of the learning process that should be related to test scores. The data are from the Early Childhood Longitudinal Study’s Kindergarten (ECLS-K) study, which is a nationally representative cohort of children. In the data, there are 9,862 students, where 6,993 students attended a pre-school program and 2,869 did not. Following 34 , we focus on the effect of not-attending a pre-school program; that is, the ATT is the impact of not attending pre-school on students who did not attend pre-school. There are 937 schools that serve as the cluster indicators. Student-level baseline covariates include sex, race, age at the start of kindergarten, height, weight, family type, family socio-economic status, income, parent’s education level, and indicator for whether the student lives with their biological mother. The outcome variable of interest is the average math test score. Here, the cluster sizes are small and range from 1 to 22 students per school. There are 131 clusters that do not contain any treated students, and 27 schools that do not contain any control students. One key difference across the different estimation strategies we consider is their ability to include schools where all or none of the students are treated. As such, one relevant question is whether dropping these clusters—and changing the estimand—produces substantially different results. Removing schools with no treated or control students leaves 8,703 students nested in 779 schools. Table 2 : Effective Sample Sizes for Five Different Estimators in the Class Size Application Estimator Effective Sample Size Unadjusted 6,993 Balancing weights with no cluster adjustment 3,172 Model-based IPW with cluster random intercepts 4,662 Hierarchical balancing weights 1,133 Mundlak balancing weights with GB constraint 595 Table 2 reports the effective sample size for \hat{\mu}_{0} for each estimator. Before weighting, there are 6,993 control students. Standard balancing weights and IPW with a cluster random intercept have large effective sample sizes of 3,172 and 4,662, respectively. At the other extreme, hierarchical and Mundlak balancing weights yield effective sample sizes that are much smaller: 1,133 and 595, respectively. In general, methods that only target global balance have a substantially higher ESS than those that also target local balance. This is not surprising, since targeting a much larger set of features typically requires more variable weights, which reduces the ESS. However, note that the Mundlak approach balances fewer features than the hierarchical balancing approach (the dimension of \psi(x,s) is 1200, versus the number of local balance features d\times K=18,696 ). Despite this, the Mundlak approach has a lower ESS. This is primarily due to the scale and choice of the hyperparameter \lambda being different between the two approaches. Figure 2 : Balance metrics for three cluster adjustment methods in the education application. Figure 2 shows covariate imbalance for the three cluster adjustment methods for covariates defined at the school level, student level, and student \times school interactions. As expected, random intercept IPW fails to balance school-level covariates and also allows for substantial local imbalance. The two balancing weight methods designed for clusters largely reduce these imbalances while also improving school-level covariate balance. The aggregated L_{2} imbalance measures reflect these qualitative comparisons. In the full data set (before removing any clusters), the unadjusted global imbalance metric is L_{2}^{\text{global}}=0.33 ; RI-IPW reduces this to L_{2}^{\text{global}}=0.14 ; both standard and hierarchical balancing weights reduce this even further to L_{2}^{\text{global}}=0.001 . Thus, balancing weight approaches produce far better global balance than RI-IPW. The corresponding unadjusted local imbalance metric is L_{2}^{\text{local}}=.26 ; hierarchical balancing weights reduces this to L_{2}^{\text{local}}=.05 , a reduction of 80%, while RI-IPW does not meaningfully change this metric at all. Thus, consistent with the results in Figure 4 , we see dramatic gains in covariate balance for balancing weights methods relative to RI-IPW. Figure 3 : Estimates for the effect of not attending a pre-school program on test scores for four different covariate adjustment strategies. Figure 3 summarizes results from the four estimation methods. Without any adjustment, students who did not attend a pre-school program had 0.5 SD lower math scores in kindergarten compared to their peers who attended preschool, a substantial difference. Adjusting for student-level covariates alone reduces this difference to 0.12 SD, with confidence intervals that exclude zero. Adjusting for cluster membership through RI-IPW returns a treatment effect of 0.25 SD. Here, the treatment effect seems rather large, likely reflecting the substantial imbalance left by RI-IPW. This estimate is nearly identical to those in 34 , which takes a different a different approach to cluster covariates, but also does not include any local balance constraints. Hierarchical balancing weights yields a point estimates around 0.07 SD, with confidence intervals that also exclude zero. Finally, the Mundlak balancing weight estimate is also 0.07 SD, but the confidence intervals do include zero, consistent with the much lower ESS reported in Table 2 . Notably, discarding smaller clusters appears to make little difference across these approaches. Thus, there are meaningful practical differences in cluster adjustment methods. RI-IPW produces much larger in magnitude than hierarchical balancing weights or Mundlak balancing weights. Given the similarity between the balancing weights estimates and the substantial imbalances in Figure 2 , this larger estimate likely reflects a failure to balance key covariates. At the same time, both hierarchical and Mundlak balancing weights produce nearly identical estimates that are much smaller than the other estimates. This stresses the importance of incorporating a local balance constraint or more fully conditioning on cluster covariates, albeit at the cost of smaller effective sample sizes. The agreement between hierarchical and Mundlak estimates is reassuring: recall that hierarchical balancing weights require only Assumption 2 , while Mundlak weights require the stronger Assumption 3 . The similarity of these estimates suggests that the exponential family assumption underlying the Mundlak approach is not grossly violated in this application—although it does not guarantee that the assumption holds exactly. 8.2 Health Services Research: Large Clusters Emergency general surgery (EGS) refers to medical emergencies where the injury is internal (e.g., a burst appendix); there are currently 51 medical conditions with this designation. EGS conditions account for more than 800,000 operations with an estimated 3-4 million patients annually in the United States ( 46 ; 18 ; 23 ; 40 ; 45 ) . Much research has focused on the comparative effectiveness of operative versus non-operative care for EGS conditions, and has found that the effectiveness of operative care varies with EGS conditions ( 28 ; 30 ; 37 ; 21 ) . Here, we focus on whether operative treatment has benefits for patients with hepato-pancreato-biliary (HPB) disorders, which are conditions that affect the liver, pancreas, and biliary system. Common HPB disorders include gallstones, cholecystitis (an inflammation of the gallbladder), and pancreatitis (an inflammation of the pancreas). We use a dataset that merges the American Medical Association (AMA) Physician Masterfile with all-payer hospital discharge claims from New York, Florida, and Pennsylvania in 2012-2013. The study population includes all patients admitted for emergency or urgent inpatient care for HPB disorders. The data includes a small set of demographic measures, such as indicators for race and ethnicity, sex, age, and insurance type. The dataset also includes measures of baseline patient frailty, represented by an indicator of severe sepsis or septic shock and pre-existing disability. There are indicators for 31 comorbidities based on Elixhauser indices ( 16 ) , which represent prior medical conditions that may complicate treatment for HPB conditions. Finally, there are three indicators for the three types of surgery used to treat HPB cases. We include these indicators variables as well to control for variation in surgery type. The primary outcome is a binary measure for the presence of an adverse event following either treatment, i.e., death or a prolonged length of stay in the hospital. In this setting, hospitals are the relevant cluster, since we are concerned that there are key unmeasured aspects of care that differ from hospital to hospital. In other words, we imagine that the ideal comparison is between patients that received surgery and those who did not within the sample hospital, which would hold constant hospital-level systemic factors. In our data, there are 143,415 patients nested within 540 hospitals. Patient population sizes within hospital vary considerably from a single HPB patient to 1,556 patients per hospital, with a mean of 265 patients. There are 10 hospitals with no treated patients and 18 hospitals in which every patient is treated. Removing these 28 hospitals removes 176 patients, or 0.1 percent of all patients. Table 5 contains the effective sample size for estimating \hat{\mu}_{0} . There are 36,786 control patients before weighting. Balancing weights with no cluster adjustment reduces the ESS by half (19,348); incorporating a random hospital intercept restores most of the ESS (31,124). Thus, targeting global balance alone has only a modest impact on the effective sample size. By contrast, additionally targeting local balance dramatically reduces ESS, as in the education example above: hierarchical and Mundlak weights have effective sample sizes of 4,775 and 6,623, respectively. Table 3 : Effective Sample Sizes for Five Different Estimators in the Surgery Application Estimator Effective Sample Size Unadjusted 36,786 Balancing weights with no cluster adjustment 19,348 Model-based IPW with cluster random intercepts 31,124 Hierarchical balancing weights 4,775 Mundlak balancing weights with GB constraint 6,623 Figure 4 shows covariate imbalance for the three cluster adjustment methods for covariates defined at the hospital level, patient level, and patient \times hospital interactions. Much like in the education application, RI-IPW fails to balance hospital-level covariates and also allows for substantial local imbalance. The two balancing weight methods designed for clusters completely reduce these imbalances while also improving hospital-level balance. As in the education example, the aggregated L_{2} imbalance measures reflect these qualitative comparisons. In the full data set (before removing any hospitals), the unadjusted global imbalance metric is L_{2}^{\text{global}}=0.22 ; RI-IPW reduces this to L_{2}^{\text{global}}=0.15 ; both standard and hierarchical balancing weights again reduce this to L_{2}^{\text{global}}=0.001 . Thus, balancing weights approaches produce far better global balance than RI-IPW. The corresponding unadjusted local imbalance metric is L_{2}^{\text{local}}=.21 ; hierarchical balancing weights reduces this to L_{2}^{\text{local}}=.02 , a reduction of 89%, while RI-IPW does not meaningfully change this metric at all. Again, we see dramatic gains in covariate balance for balancing weights methods relative to RI-IPW. Figure 4 : Balance metrics for three cluster adjustment methods in the surgery application. Figure 5 shows the point estimates and 95% confidence intervals for the effect of surgery. The unadjusted estimate indicates that surgical care lowers the risk of an adverse event by almost 9%, although this is likely biased if healthier patients are selected for surgery. Indeed adjusting for patient covariates alone flips the sign and indicates that surgical care actually increases the risk of an adverse event by around 3%, with confidence intervals bounded away from zero. Next, RI-IPW yields an estimate that is quite close to the unadjusted estimate. However, as we observed above, RI-IPW fails to achieve good covariate balance and this estimate likely reflects substantial bias. Hierarchical balancing weights, by contrast, estimate an effect that is essentially zero—and is much closer to the balancing weights estimate that does not include cluster adjustments. Finally, adjusting for hospital-level sufficient statistics in the Mundlak estimator similarly yields a small estimate that is not statistically significant. Figure 5 : Estimates for the effect of surgery on adverse events for five different covariate adjustment strategies. Comparing these estimates, RI-IPW likely yields a poor estimate of the causal effect, with large covariate imbalance and with estimates that are clear outliers relatively to the other methods; this is consistent with both the simulation evidence and the education application. Next, the two cluster adjustments methods produce similar estimates, which indicates that accounting for the hospital context is likely an important part of the design. We find that there appears to be little benefit to surgical treatment for HPB conditions, once we account for the care environment. By contrast, if we only balance patient covariates, we would find that surgery appears to harm patients. As in the education application, the agreement between hierarchical and Mundlak balancing weights provides some reassurance about Assumption 3 . However, researchers should note that this agreement does not validate the stronger assumption; it merely suggests that the sufficient statistics capture enough cluster-level variation to produce similar estimates. In settings where hierarchical methods are feasible—as they are here given the larger cluster sizes—researchers may prefer hierarchical balancing weights precisely because they avoid the additional exponential family assumption. 9 Discussion Taken together, our results suggest that researchers should generally prefer balancing weights approaches to traditional, model-based IPW with a hierarchical propensity score. RI-IPW is particularly problematic when treatment assignment varies within clusters. When feasible, researchers should also prioritize methods that target both global and local balance, such as hierarchical and Mundlak balancing weights. The latter are especially promising when cluster-level covariates are available and when cluster sizes are small. However, researchers should recognize that Mundlak methods require the stronger Assumption 3 , which posits that cluster-level sufficient statistics fully capture cluster-level confounding. When cluster sizes are large enough to permit direct conditioning on cluster membership, hierarchical balancing weights under the weaker Assumption 2 may be preferable, as they avoid the additional exponential family structure. In general, we recommend comparing estimates across multiple methods (e.g., no cluster adjustment, hierarchical balancing weights, and Mundlak balancing weights). If cluster adjustment is important, estimates from methods that target local balance should be similar to each other and potentially different from estimates that ignore clusters or only target global balance. Agreement between hierarchical and Mundlak balancing weights provides some reassurance that Assumption 3 is not grossly violated; disagreement may indicate that the sufficient statistics do not fully capture cluster-level confounding or that the hierarchical approach was unable to achieve good local balance. There are several important directions for future research. First, our paper focuses on weighting methods and only briefly addresses bias correction with an outcome model. Understanding the interplay between weighting and outcome modeling is important for this setting. Moreover, our focus has been on observational studies with a clustered data structure. 2 , however, connect this same setup to the panel data setting; similarly adapting our framework to panel data estimation is a promising direction. Another extension is to move beyond within-study average treatment effects, such as the ATT, to heterogeneous treatment effects and transportability. Finally, we focus on only two levels of hierarchy here. While this is by far the most common setting, cases with multiple levels, such as students nested within classrooms nested within schools, introduce many additional complications. Appendix A Additional Empirical Results Here, we present additional empirical results omitted from the main paper. We include two additional sets of results. First, we report the results from the AvTO Mundlak weighting approach. Second, we report treatment effect estimates that include augmentation from an outcome model. In both cases, the results are unchanged from those reported in the main text. A.1 Education: Small Clusters Table 4 : Effective Sample Sizes for Five Different Estimators in the Education Application Estimator Effective Sample Size Unadjusted 6,993 Balancing weights with no cluster adjustment 3,173 Model-based IPW with cluster random intercepts 4,662 Hierarchical balancing weights 1,133 Mundlak balancing weights with GB constraint 595 Mundlak balancing weights with AvTO constraint 609 Figure 6 : Estimates for the effect of not attending a pre-school program on test scores for five different covariate adjustment strategies. Figure 7 : Estimates for the effect of not attending a pre-school program on test scores for five different covariate adjustment strategies with and without outcome model augmentation/bias correction. A.2 Surgery Table 5 : Effective Sample Sizes for Five Different Estimators in the Surgery Application Estimator Effective Sample Size Unadjusted 36,786 Balancing weights with no cluster adjustment 19,348 Model-based IPW with cluster random intercepts 31,124 Hierarchical balancing weights 4,775 Mundlak balancing weights with GB constraint 6,623 Mundlak balancing weights with AvTO constraint 5,677 Figure 8 : Estimates for the effect of surgery on adverse events for five different covariate adjustment strategies. Figure 9 : Estimates for the effect of surgery on adverse events for five different covariate adjustment strategies with and without outcome model augmentation/bias correction.