Bayesian Hierarchical Invariant Prediction

Francisco Madaleno, Pernille Julie Viuff Sand, Francisco C. Pereira, Sergio Hernan Garrido Mejia · 2025

Read Bayesian Hierarchical Invariant Prediction by Francisco Madaleno, Pernille Julie Viuff Sand, Francisco C. Pereira, Sergio Hernan Garrido Mejia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11211.txt.

Introduction

Francisco Madaleno and three co-authors propose a new method in a 2025 arXiv paper on finding causes that stay stable across settings. Their method, called Bayesian Hierarchical Invariant Prediction, reworks an older approach known as Invariant Causal Prediction. It uses layered models to test whether a causal link holds when data differ.

Because the method is Bayesian, it can also use prior knowledge. The layered setup makes it faster to run when there are many possible predictors. The team tests it on both made-up and real data sets. They present it as a useful option beside the older method and related tools.

Catalog record

CREATORFrancisco Madaleno, Pernille Julie Viuff Sand, Francisco C. Pereira, Sergio Hernan Garrido Mejia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11211

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bayesian Hierarchical Invariant Prediction Francisco Madaleno, Pernille Julie Viuff Sand, Francisco C. Pereira, Sergio Hernan Garrido Mejia Abstract We propose Bayesian Hierarchical Invariant Prediction (BHIP) reframing Invariant Causal Prediction (ICP) through the lens of Hierarchical Bayes. We leverage the hierarchical structure to explicitly test invariance of causal mechanisms under heterogeneous data, resulting in improved computational scalability for a larger number of predictors compared to ICP. Moreover, given its Bayesian nature BHIP enables the use of prior information. We evaluate BHIP on both synthetic and real-world datasets, demonstrating its potential as an alternative inference method to ICP and related methods. keywords causality, invariant causal prediction, hierarchical bayes 1 Introduction Heterogeneous data is ubiquitous in real-world applications, spanning fields such as medicine, transportation, and environmental science. The critical observation of this paper is that the heterogeneous data forming the basis of Bayesian Hierarchical Models (BHM) corresponds to “environment data” described in Invariant Causal Prediction (ICP) ( Peters et al., 2016 ) . Both perspectives acknowledge that real-world data often arises from distinct underlying processes or ‘environments,’ underscoring the need for robust models that account for these differences and ultimately improve out-of-distribution performance. For example, data collected across hospitals, regions, or time periods often exhibit distribution shifts: patient populations and treatment protocols may differ across clinics, and travel behavior may differ across cities or seasons. Without specific structural assumptions observational data can only allow inferring a Directed Acyclic Graph (DAG) model up to the Markov Equivalence Class ( Pearl, 2000 ) . ICP identifies causal predictors by leveraging invariance properties across environments, exhaustively employing conditional independence tests to find sets of predictors for which the target variable is independent of the environment when conditioned on those predictors. As most statistical methods, ICP is sample-size sensitive, but also computationally complex since it involves testing for invariance across all possible subsets of predictors, 2^{d} where d is the number of predictors. Additionally, ICP is designed to be conservative and minimize Type I errors which often leads to low power ( Peters et al., 2016 ) . Addressing these limitations, we introduce Bayesian Hierarchical Invariant Prediction (BHIP), which models the variability of predictors’ effects across environments, quantifies uncertainty, and allows for the incorporation of priors (which we explore in Appendix E with the inclusion of sparsity inducing priors), enhancing flexibility and overall performance compared to frequentist approaches like ICP. While BHIP utilizes established Bayesian techniques, namely hierarchical modeling, its novelty lies not in the individual components themselves, but in their specific synthesis and application to probabilistically test causal invariance for parent discovery. Specifically, this work contributes a unified probabilistic framework for invariant prediction. Leveraging its hierarchical structure and Bayesian nature, it simultaneously accounts for environmental heterogeneity, allows variable selection (e.g., through sparsity-inducing priors), quantifies uncertainty in estimated effects and, crucially, enables probabilistic invariance testing directly within the model’s posterior analysis. By repurposing the standard hierarchical modeling structure and combining it with specific posterior decision rules, BHIP directly identifies invariant parents in heterogeneous data. This targeted and integrated Bayesian approach represents a significant step beyond simply applying standard Bayesian methods or relying solely on frequentist invariance testing. Code is available at GitHub repository 1 1 1 https://github.com/fmfsa/bhip . 1.1 Related Work The problem of causal feature selection or causal discovery on a specific target variable has been studied as an alternative to the more difficult problem of full causal discovery. Under Pearl’s view of Causality ( Pearl, 2000 ) , ICP laid down the foundations for causal feature selection using the idea of invariance. What differentiated ICP from other methods for causal discovery using both interventional and observational data is that ICP does not require the analyst to know where the interventions are performed, but instead only to know from which setting a particular data point comes. This powerful idea was later extended to non-linear models ( Heinze-Deml et al., 2018 ) , dynamical systems ( Pfister et al., 2019a ) , time series data ( Pfister et al., 2019b ) , spatio-temporal data ( Christiansen et al., 2022 ) and different outcome models ( Kook et al., 2024 ) . Moreover, the invariant principle has been also used together with adjacent machine learning methods: active learning ( Gamella and Heinze-Deml, 2020 ) , to find the causal features with experimental data efficiently; and reinforcement learning ( Saengkyongam et al., 2023 ) , for policy learning. Bayesian inference has also been used for causal discovery, ( Heckerman et al., 2006 ) being one of the earliest examples with Monte Carlo inference on causal graphs, all the way to recent advances, such as ( Hägele et al., 2023 ) who use differentiable methods to infer causal structure in a Bayesian way. The theoretical foundations of the Bayesian method have also helped gain insights in causality using the invariance principle in exchangeable data ( Guo et al., 2024a ; Guo et al., 2024b ) , and understanding causality in the context of hierarchical models ( Weinstein and Blei, 2024 ) . The relation between hierarchical Bayes and causal discovery is long-standing. ( Gelman and Imbens, 2013 ) talk about the differences between asking about effects of causes (causal inference) and causes of effects (causal discovery) and mention how bayesian, and specifically hierarchical, models can be used for the latter (they present ( Manton et al., 1989 ) as an example). Concurrently, Wu et al. (2025) develop a Bayesian invariant prediction model that, alike this work, casts ICP-style invariance as posterior inference over invariant features, but differs by encoding the invariant set through a latent feature-selection variable in a joint generative model with consistency and variational-inference guarantees, rather than via hierarchical partial pooling on regression coefficients as in BHIP. 2 Background 2.1 Graphical Models and Causal Inference In this paper we use SCMs ( Peters et al., 2017 ) to represent causal relations. Definition 1 (Structural Causal Model (SCM)) . A D- dimensional SCM is a tuple \mathcal{M}:=(\mathbb{S},P_{\varepsilon}) consisting of a set \mathbb{S} of structural assignments \displaystyle X_{d}:=f_{d}(\text{PA}(X_{d}),\varepsilon_{d})),\text{ for }d=1,\dots,D, (1) where \text{PA}(X_{d})\subseteq\{X_{1},\dots,X_{D}\}\setminus\{X_{d}\} are called the parents or causes of X_{d} , and a joint distribution P_{\varepsilon}=P_{\varepsilon_{1},\dots,\varepsilon_{D}} over noise variables that we assume to be independent. We can obtain a causal graph \mathcal{G} from the SCM by drawing a vertex for each X_{d} and a directed edge from each vertex X_{i}\in\text{PA}(X_{d}) to X_{d} . We assume the obtained causal graph is a DAG. Likewise, even though the causal mechanisms are deterministic functions of its inputs, we obtain a distribution of each of the variables of our system, P_{X_{d}} , by considering the pushforward distribution of \text{PA}(X_{d}) and \varepsilon_{d} . We exploit the independent ( Janzing and Schölkopf, 2010 ) and invariant ( Aldrich, 1989 ; Peters et al., 2016 ) nature of causal mechanisms to identify the causes of a target variable, Y , using heterogeneous data. 2.2 Invariant Causal Prediction Traditionally, Machine Learning methods assume data is independent and identically distributed (i.i.d) , unless specified by a particular structure like time series or graph models. Researchers in causality have found ways of exploiting heterogeneous data to do causal discovery ( Cooper and Yoo, 1999 ; Mooij et al., 2020 ; Brouillard et al., 2020 ; Peters et al., 2016

Read the complete text