A Partial Initialization Strategy to Mitigate the Overfitting Problem in CATE Estimation with Hidden Confounding

Chuan Zhou, Yaxuan Li, Chunyuan Zheng, Haiteng Zhang, Haoxuan Li, Mingming Gong · 2025

Read A Partial Initialization Strategy to Mitigate the Overfitting Problem in CATE Estimation with Hidden Confounding by Chuan Zhou, Yaxuan Li, Chunyuan Zheng, Haiteng Zhang, Haoxuan Li, Mingming Gong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-08888.txt.

Introduction

Chuan Zhou and five co-authors gauge who gains from a treatment when hidden confounders mar data drawn from records of real life. Gauges of the conditional average treatment effect guide calls in commerce, health care, and economics. Standard tools take it that there is no hidden confounding. That is a claim no test can check, and it can void what the tools find.

Trials with random assignment dodge confounding but are small, which invites overfitting. Their two-stage frame first learns a way to show covariates from large sets of records, to predict counterfactual outcomes. A second stage, fine-tuned on trial data, adds a further form that adjusts for hidden confounding. Part of the prediction heads carry over in place of training from scratch. Tests showed the approach beat its rivals when confounding was hidden.

Catalog record

CREATORChuan Zhou, Yaxuan Li, Chunyuan Zheng, Haiteng Zhang, Haoxuan Li, Mingming Gong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-08888

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Partial Initialization Strategy to Mitigate the Overfitting Problem in CATE Estimation with Hidden Confounding Chuan Zhou, Yaxuan Li, Chunyuan Zheng, Haiteng Zhang, Haoxuan Li, Mingming Gong Abstract . Estimating the conditional average treatment effect (CATE) from observational data plays a crucial role in areas such as e-commerce, healthcare, and economics. Existing studies mainly rely on the strong ignorability assumption that there are no hidden confounders, whose existence cannot be tested from observational data and can invalidate any causal conclusion. In contrast, data collected from randomized controlled trials (RCT) do not suffer from confounding but are usually limited by a small sample size. To avoid overfitting caused by the small-scale RCT data, we propose a novel two-stage pretraining-finetuning (TSPF) framework with a partial parameter initialization strategy to estimate the CATE in the presence of hidden confounding. In the first stage, a foundational representation of covariates is trained to estimate counterfactual outcomes through large-scale observational data. In the second stage, we propose to train an augmented representation of the covariates, which is concatenated with the foundational representation obtained in the first stage to adjust for the hidden confounding. Rather than training a separate network from scratch, part of the prediction heads are initialized from the first stage. The superiority of our approach is validated on two datasets with extensive experiments. DOI: XXXXXXX.XXXXXXX Conference: Causal Inference and Machine Learning in Practice; August 2024; Barcelona, Spain CMLKDD2024: KDD 2024 Workshop, Aug 25–29, 2024, Barcelona, Spain ISBN: 978-1-4503-XXXX-X/18/06 CCS: Computing methodologies Machine learning approaches 1. Introduction The conditional average treatment effect (CATE) is the average causal effect of a treatment or an intervention on the outcome of interest given the covariates ( Pearl, 2009 ) , which plays an important role in diverse fields, such as trustworthy artificial intelligence ( Li et al., 2023b ) , electronic commerce ( Wu et al., 2022 ) , healthcare ( Robins and Hernán, 2016 ) , and economics ( Huynh et al., 2016 ) . In e-commerce, the platforms desire to predict how recommending a specific product to a particular user affects the probability of purchase ( Li et al., 2022 ) , and thereby influence the total profit. In healthcare, doctors assess the potential outcome for different patient groups when receiving a certain treatment ( Corrao et al., 2012 ) for precision medicine. Similarly in economics, the policymakers evaluate how much a job training program will raise employment opportunities for certain groups of unemployed individuals ( Britto et al., 2022 ) . To enhance the accuracy of CATE estimation, representation-based learning approaches have gathered increasing attention due to their impressive performance ( Johansson et al., 2016 ; Li et al., 2024b ; Zhu et al., 2024 ; Wang et al., 2024 ) . These approaches focus on generating covariate representations, to mitigate confounding bias by minimizing distributional discrepancies between the treatment and control groups. Previous approaches have developed substantial theory and explored extensive practice to obtain such representations. For instance, some of them use integral probability metric (IPM) for regularization ( Johansson et al., 2016 ) , while a few approaches emphasize local similarity preservation ( Yao et al., 2018 ) , targeted learning ( Zhang et al., 2020 ) , or view the problem from new perspectives like optimal transport ( Wang et al., 2024 ) and out-of-distribution ( Li et al., 2024a ) . However, the aforementioned methods may ignore hidden confounding, which is very common in real-world scenarios ( Li et al., 2024c ) . In our e-commerce example, the financial status of users might be sensitive and difficult to collect ( Li et al., 2023a ) . In the healthcare case, the personal lifestyles of patients are difficult to obtain ( Charpignon et al., 2022 ) . For the example in economics, personal working status is difficult to measure ( Xu and Li, 2020 ) . These hidden variables can affect treatment and outcome simultaneously, which causes confounding bias in the estimation of causal effects. Therefore, proposing methods to account for the confounding bias is crucial to accurately estimate CATE. To address the hidden confounding problem, one category of mainstream methods can rely only on large-scale observational (OBS) data, including sensitivity analysis ( Imbens, 2003 ; Dorn et al., 2024 ) , front-door adjustment methods ( Fulcher et al., 2020 ) , and instrumental variables methods ( Joshua D. Angrist and Rubin, 1996 ) . These methods require additional strong assumptions that cannot be tested from the data and raise concerns if these assumptions are violated ( Hartwig et al., 2023 ; Kong et al., 2025 ) . Compared to the OBS data, randomized controlled trial (RCT) data are considered as the gold standard for causal effect estimation ( Prosperi et al., 2020 ) . However, practical challenges such as high costs and ethical concerns may make the collection of RCT data difficult ( Zabor et al., 2020 ; Bédécarrats et al., 2020 ) , resulting in limited sample sizes. Due to the small size, it is impractical to directly train causal effect prediction models on RCT data alone ( Hoogland et al., 2021 ) . Therefore, it is necessary to find an effective method to combine small-scale RCT data with large-scale OBS data. However, previous methods either rely on untestable strong assumptions ( Kallus et al., 2018 ; Hatt et al., 2022b ) or suffer from the risk of overfitting due to limited RCT sample sizes ( Colnet et al., 2024 ; Wu and Yang, 2022 ) . In this paper, we introduce a two-stage pretraining-finetuning (TSPF) framework with a partial initialization strategy for CATE estimation under hidden confounding. Our approach leverages large-scale OBS data to train a foundational representation of covariates and then uses relatively small-scale RCT data to adjust the bias in the representation learned from OBS data. We then train an unbiased prediction model using this adjusted representation. In the second stage, we introduce an additional module that ensures stronger representation ability compared to the methods that use RCT data to estimate the residuals and initialize part of the prediction heads with parameters from the first stage. The contributions of this paper are summarized as follows. • We present the TSPF framework that includes a partial initialization strategy for CATE estimation, tackling hidden confounding by using a small amount of unconfounded RCT data to adjust the representations learned from OBS data. • The proposed framework does not rely on the linear and additive generation assumptions, and can flexibly adjust its model structure according to the sample size of RCT data, thus mitigating the over-fitting problem. • Experiments conducted on the semi-synthetic IHDP and Jobs datasets demonstrate the effectiveness of our approach. 2. Preliminaries 2.1. Problem Setup We consider two independent data sources taken from the same target population: one from OBS and the other from RCT. Each individual in the OBS or RCT study is an observation of (X,Y,T,G) , a random tuple with distribution P . For the i -th individual, the observation comprises d -dimensional covariates X_{i}\in\mathcal{X}\subseteq\mathbb{R}^{d} , the observed outcome Y_{i} , the assigned binary treatments T_{i}\in\{0,1\} ( T_{i}=0 for the controlled and T_{i}=1 for the treated individuals) and G_{i} denoting participation in the OBS

Read the complete text