Long-Term Individual Causal Effect Estimation via Identifiable Latent Representation Learning

Ruichu Cai, Junjie Wan, Weilin Chen, Zeqin Yang, Zijian Li, Peng Zhen, Jiecheng Guo · 2025

Read Long-Term Individual Causal Effect Estimation via Identifiable Latent Representation Learning by Ruichu Cai, Junjie Wan, Weilin Chen, Zeqin Yang, Zijian Li, Peng Zhen, Jiecheng Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05192.txt.

Introduction

Trials end too soon to show a treatment's long-term effect, so analysts marry short trial data to long observational records, where hidden confounders lurk. Ruichu Cai and six co-authors estimate long-term individual effects in 2025 without leaning on the field's usual idealized assumptions.

Older methods assume the hidden factors away or force them into a fixed bias shape, which real data rarely honor. The new approach instead uses natural variety in the data, such as several sources, to identify the hidden factors through learned latent representations. The authors prove that identification is possible on those terms, then build an estimator on top. Tests on synthetic and semi-synthetic data sets show the method holding up where the assumptions of rivals break.

Catalog record

CREATORRuichu Cai, Junjie Wan, Weilin Chen, Zeqin Yang, Zijian Li, Peng Zhen, Jiecheng Guo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-05192

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Long-Term Individual Causal Effect Estimation via Identifiable Latent Representation Learning Ruichu Cai, Junjie Wan, Weilin Chen, Zeqin Yang, Zijian Li, Peng Zhen, Jiecheng Guo Abstract Estimating long-term causal effects by combining long-term observational and short-term experimental data is a crucial but challenging problem in many real-world scenarios. In existing methods, several ideal assumptions, e.g. latent unconfoundedness assumption or additive equi-confounding bias assumption, are proposed to address the latent confounder problem raised by the observational data. However, in real-world applications, these assumptions are typically violated which limits their practical effectiveness. In this paper, we tackle the problem of estimating the long-term individual causal effects without the aforementioned assumptions. Specifically, we propose to utilize the natural heterogeneity of data, such as data from multiple sources, to identify latent confounders, thereby significantly avoiding reliance on idealized assumptions. Practically, we devise a latent representation learning-based estimator of long-term causal effects. Theoretically, we establish the identifiability of latent confounders, with which we further achieve long-term effect identification. Extensive experimental studies, conducted on multiple synthetic and semi-synthetic datasets, demonstrate the effectiveness of our proposed method. 1 Introduction Estimating long-term causal effects is of increasing importance in many domains, such as healthcare, public education, marketing, and public policy ( Hohnhold et al. 2015 ; Chetty et al. 2011 ; Fleming et al. 1994 ; Zheng et al. 2025 ) . In such long-term scenarios, it is usually difficult to conduct randomized control experiments to estimate the causal effects. Hence, a lot of researchers resort to the more easily accessible long-term observations. However, methods based on observational data still suffer from the latent confounding bias problem. Therefore, combining observational data and experimental data has emerged as a promising solution for estimating long-term causal effects ( Imbens et al. 2024 ; Ghassami et al. 2022 ; Hu et al. 2022 ) . (a) (b) (c) Figure 1: Three causal graphs in long-term scenarios with X being the pre-treatment variables, Y being the long-term outcome, Z being the latent confounders, S being short-term outcome, U being the auxiliary variable, and W being the treatment. White nodes denote the observed variables and grey nodes denote the unobserved variables. The dashed edges exist in the observational data but are absent in the experimental data. The dashed node Y means Y can be observed in observational data but not in experimental data. Specifically, Fig. 1a shows the causal graph satisfying the latent confoundedness assumption ( Athey et al. 2020 ) . Fig. 1b shows the causal graph satisfying the equi-confounding bias assumption ( Ghassami et al. 2022 ) , where the blue arrows in Fig. 1b indicate the equal confounding bias. Fig. 1c shows the causal graph of our setting. Existing data combination-based methods estimate long-term effects mainly based on the so-called surrogate . As shown in Fig. 1 , the surrogate S is the short-term outcome, serving as the supplement or replacement for the long-term outcome Y in observational data. However, the unconfoundedness assumption is usually violated in such observational data due to the existence of latent confounders Z . As a replacement for unconfoundedness assumption, Athey et al. 2020 propose an assumption named latent unconfoundedness, i.e., Y(w)\perp W|X,S(w) on observational data, implicitly indicating the latent confounders Z cannot affect long-term outcome Y as illustrated in Fig. 1a . Alternatively, to relax the unconfoundedness assumption, Ghassami et al. 2022 introduces the (conditional) additive equi-confounding bias assumption, i.e., the magnitude of the confounding bias for the short-term and the long-term potential outcome variables are the same, as illustrated in Fig. 1b . Existing methods, however, encounter a key challenge : the ideal assumptions are usually violated in real-world applications, including both the latent unconfoundedness and additive equi-confounding bias assumptions, which limit their practical effectiveness. For example, in studying the effect of driver income (treatment W ) on long-term retention (outcome Y ) in a ride-hailing platform, driver characteristics (pre-treatment variable X ) act as observed confounders affecting both income and retention. However, the drivers’ household expenses (latent confounders Z ) may also affect drivers’ long-term retention Y , violating the latent unconfoundedness assumption. Similarly, the additive equi-confounding bias assumption may be violated since household expenses can influence short- and long-term retention differently, i.e., the confounding bias varies over time rather than remaining constant. Therefore, the strong assumptions in existing methods still significantly limit their applicability. To address the above challenge, we aim to develop a method without the above assumptions to estimate the individual long-term causal effects as shown in Fig. A.1 . Specifically, instead of assuming latent unconfoundedness or equi-confounding bias, we explore the identifiability of latent confounders Z to estimate long-term causal effects. To identify latent confounder Z , we resort to an additional auxiliary variable U , which is easily accessible from our readily available prior knowledge, such as the natural heterogeneity of data in real-world applications. Recall the aforementioned drivers’ income study example, the data are usually collected from various cities, and the indicator variable of the city can be directly taken as the auxiliary variable. Leveraging the identifiability of Z , we establish the causal effect identification result and propose the corresponding latent representation learning-based estimator for long-term individual causal effects. Overall, our contributions can be summarized as follows: • We focus on a more general setting for estimating long-term causal effects, as shown in Fig. 1c . As shown in Fig. 1c , the assumed causal graph in our paper is a complete graph, and the causal graphs in existing work ( Athey et al. 2020 ; Ghassami et al. 2022 ) can be seen as our special cases. • We theoretically achieve the identifiability of latent confounders. Leveraging the identifiability result, we further establish the identification of long-term individual effects. • We devise a latent representation learning-based estimator for effect estimation. The effectiveness of our estimator is verified on five synthetic and two real-world datasets. 2 Related Works Variational Auto-encoders for Causal Inference Variational Auto-encoder (VAE) ( Kingma and Welling 2014 ) is a powerful tool to capture latent structure in different kinds of applications, e.g., image processing ( Gregor et al. 2015 ) and time-series ( Chung et al. 2015 ; Cai et al. 2025 ) . In causal inference, VAE is used to recover unobserved variables to achieve the identification and estimation of the effects. Without unconfoundedness assumption, CEVAE ( Louizos et al. 2017 ) assumes that latent confounders can be recovered by their proxies and applies VAE to learn confounders. As a follow-up work, TEDVAE ( Zhang et al. 2021 ) and DMAVAE ( Xu et al. 2023 ) decouples the learned latent confounders into several factors to achieve a more accurate estimation of treatment effects in different settings. With the recent development of VAE, nonlinear independent component analysis theory ( Hyvarinen and Morioka 2016 ) enables the identifiability of recovered variables, e.g., iVAE Khemakhem et al. 2020 and SIG ( Li et al. 2023 ) . CFDiVAE ( Xu et al. 2024 ) apply iVAE to recover the front-door adjustment variable, achieving effect identification under the front-door criterion ( Pearl 2009 ) . \beta -Intact-VAE ( Wu and Fukumizu 2022 ) utilizes iVAE to recover prognostic scores to estimate effects under a

Read the complete text