Beyond Reweighting: On the Predictive Role of Covariate Shift in Effect Generalization

Ying Jin, Naoki Egami, Dominik Rothenhäusler · 2024

Read Beyond Reweighting: On the Predictive Role of Covariate Shift in Effect Generalization by Ying Jin, Naoki Egami, Dominik Rothenhäusler (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08869.txt.

Introduction

Ying Jin, Naoki Egami, and Dominik Rothenhäusler study how study results travel between settings in 2024. When a trial's people differ from a target group, analysts adjust for the traits they can see. The catch: gaps in unseen traits often remain, and seeing-trait gaps alone rarely explain the full shift.

Across 680 studies at sixty-five sites, the team found a pattern. The unseen gap's strength could often be bounded by the seen gap's strength, once both were put on their new standard scale. A random shift model explains why. Using the bound sharpens error bars for target estimates when some data are missing.

Catalog record

CREATORYing Jin, Naoki Egami, Dominik Rothenhäusler
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE129 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08869

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Reweighting: On the Predictive Role of Covariate Shift in Effect Generalization Ying Jin, Naoki Egami, Dominik Rothenhäusler Abstract Many existing approaches to generalizing statistical inference amidst distribution shift operate under the covariate shift assumption, which posits that the conditional distribution of unobserved variables given observable ones is invariant across populations. However, recent empirical investigations have demonstrated that adjusting for shift in observed variables (covariate shift) is often insufficient for generalization. In other words, covariate shift does not typically “explain away” the distribution shift between settings. As such, addressing the unknown yet non-negligible shift in the unobserved variables given observed ones (conditional shift) is crucial for generalizable inference. In this paper, we present a series of empirical evidence from two large-scale multi-site replication studies to support a new role of covariate shift in “predicting” the strength of the unknown conditional shift. Analyzing 680 studies across 65 sites, we find that even though the conditional shift is non-negligible, its strength can often be bounded by that of the observable covariate shift. However, this pattern only emerges when the two sources of shifts are quantified by our proposed standardized, “pivotal” measures. We then interpret this phenomenon by connecting it to similar patterns that can be theoretically derived from a random distribution shift model. Finally, we demonstrate that exploiting the predictive role of covariate shift leads to reliable and efficient uncertainty quantification for target estimates in generalization tasks with partially observed data. Overall, our empirical and theoretical analyses suggest a new way to approach the problem of distributional shift, generalizability, and external validity. Keywords: Generalizability, external validity, distribution shift, replication studies. 1 Introduction Distribution shift is a central issue in generalizing statistical evidence from an observed (source) population to a new, at most partially observed (target) population, with significant implications in many domains. For instance, in the medical and social sciences, researchers/policymakers seek to leverage existing randomized control trials (RCTs) to estimate the treatment effect on a new cohort to guide clinical decisions or policy making ( Shadish et al.,, 2002 ; Hotz et al.,, 2005 ; Imai et al.,, 2008 ; Cole and Stuart,, 2010 ; Tipton,, 2013 ; Bareinboim and Pearl,, 2016 ; Deaton and Cartwright,, 2018 ) . However, the challenge lies in whether statistical methods can capture the changes between populations to produce credible predictions of target effects. To address the generalizability question, many statistical methods operate under assumptions positing that observed variables capture all distributional differences between populations. These assumptions can often be described as covariate shift , that is, the distribution of covariates observed in both populations can change, while the conditional distribution of the outcomes (unobserved in the target population) given the observed covariates remains invariant. For example, the distribution of age, gender, and education can differ across populations (e.g., due to convenience sampling), but the conditional treatment effect is the same for individuals with the same covariate profiles. Under this common assumption, adjusting for shift in the observed covariates, either by reweighting based on density ratios or estimating the heterogeneous covariate-outcome relationship ( Stuart et al.,, 2011 ; Tipton et al.,, 2014 ; Miratrix et al.,, 2018 ; Dahabreh et al.,, 2019 ; Egami and Hartman,, 2023 ) , is sufficient for unbiased estimation of the target parameters. This common approach highlights the role of covariate shift in explaining away the distribution shift. Given its popularity, a series of recent papers ( Cai et al.,, 2023 ; Jin et al.,, 2023 ; Lu et al.,, 2023 ) have empirically evaluated the performance of generalization estimators based on the covariate shift assumption by comparing them against experimental benchmark estimates. Although each paper focuses on different domains, a common yet somewhat surprising finding is that observed covariate shift often can only explain a small proportion of the distributional shift in real-world applications. This implies two pessimistic messages: (1) adjusting for observed covariate shift may be insufficient for generalization, and (2) the remaining, unobserved conditional shift (i.e., shift in the conditional distribution of the outcomes given the observed covariates) is “larger” than the observed covariate shift. As such, it remains unclear how the conditional shift may be addressed for effect generalization in practice even in well-controlled settings. 1.1 This work: the predictive role of covariate shift In this paper, we introduce a different role of covariate shift in predicting the unknown shift in the conditional distribution for generalization (Figure 1 ). The distribution shift between the source and target populations consists of the observed covariate shift and unobserved conditional shift, the latter being a key challenge in a generalization task. In contrast to existing approaches that either (i) assume there is no conditional shift, or (ii) establish worst-case bounds based on adversarial shift in the conditional distribution, we argue that the strength of covariate shift can bound that of the unknown conditional shift. Exploiting this bounding relationship is useful in effect generalization with improved validity and efficiency. Figure 1 : Overview of the problem and our approach: Effect generalization from source and target populations needs to address the distribution shift consisting of the observed covariate shift and unobserved conditional shift. We argue a novel predictive role of covariate shift in bounding the strength of unknown conditional shift, which is supported by our empirical findings and leads to reliable and efficient generalization. Our proposal is supported by empirical evidence from two well-known, large-scale multi-site replication projects—the Pipeline project ( Schweinsberg et al.,, 2016 ) and the Many Labs 1 project ( Klein et al.,, 2014 ) ---from the social sciences, analyzing a total of 680 studies across 65 sites examining 25 hypotheses. 1 1 1 Note that not all sites examine all hypotheses. To ensure faithful evaluation, since we have no access to the underlying population parameters, we build prediction intervals—based on various distribution shift assumptions—for estimators in target populations (including our proposed ones built upon empirical findings), and use their empirical coverage to examine the plausibility of the assumptions they are based upon. Figure 2 previews our empirical results. Figure 2 : Preview of results. [Left] Insufficient explanatory role of covariate shift: Empirical coverage of prediction intervals based on i.i.d. assumption (grey) and covariate shift assumption (green and purple), showing covariate shift cannot explain away distribution shift across sites. [Right] Reliable and efficient effect generalization based on the predictive role of covariate shift: Empirical coverage of prediction intervals based on i.i.d. assumption (grey), worst-case bounds (dark blue), and our method with the belief that conditional shift is bounded by covariate shift (red) or with knowledge of their relative strength (yellow). We begin by examining common approaches that either ignore distribution shift or assume covariate shift (Section 2.3 ). In the two replication projects, the explanatory role of covariate shift is limited as evident from the low coverage of prediction intervals, complementing existing work that either examine pairs of studies ( Jin et al.,, 2023 ) or mean squared errors ( Lu et al.,, 2023 ; Kern et al.,, 2016 ) . As shown in Panel (a) of Figure 2 , even for controlled multi-site replication studies, distribution shifts across sites are not negligible (methods that assume no distributional shift ( IID ) do not achieve valid coverage). Furthermore,

Read the complete text