Use What You Know: Causal Foundation Models with Partial Graphs Arik Reuter, Anish Dhir, Cristiana Diaconu, Jake Robertson, Ole Ossen, Frank Hutter, Adrian Weller, Mark van der Wilk, Bernhard Schölkopf, [0.5em] University of Cambridge, Cambridge, United Kingdom Abstract Estimating causal quantities traditionally relies on bespoke estimators tailored to specific assumptions. Recently proposed Causal Foundation Models (CFMs) promise a more unified approach by amortising causal discovery and inference in a single step. However, in their current state, they do not allow for the incorporation of any domain knowledge, which can lead to suboptimal predictions. We bridge this gap by introducing methods to condition CFMs on causal information, such as the causal graph or more readily available ancestral information. When access to complete causal graph information is too strict a requirement, our approach also effectively leverages partial causal information. We systematically evaluate conditioning strategies and find that injecting learnable biases into the attention mechanism is the most effective method to utilise full and partial causal information. Our experiments show that this conditioning allows a general-purpose CFM to match the performance of specialised models trained on specific causal structures. Overall, our approach addresses a central hurdle on the path towards all-in-one causal foundation models: the capability to answer causal queries in a data-driven manner while effectively leveraging any amount of domain expertise. (a) (b) Figure 1: Causal Foundation Models based on PFNs for predicting the effect of causal interventions: A model takes as input observational data \mathcal{D} and a query-point comprising an intervention do(t) together with a feature vector \bm{x} . The model, implemented as a transformer, outputs the posterior of the causal effect p(y|\text{do}(t),\bm{x},\mathcal{D}) . This can be seen as the model marginalising over causal structures (SCMs) \psi , based on the posterior probability that \psi could have generated the observational data \mathcal{D} . (a) The model can only leverage observational data to compute its posterior belief p(\psi|\mathcal{D}) over possible causal structures (visualized as the region shaded in blue). When using this posterior to compute the posterior predictive \smash{p(y|\text{do}(t),\bm{x},\mathcal{D})=\int p(y|t,\bm{x},\psi_{do(t)})p(\psi|\mathcal{D})d\psi} , this leads to high uncertainty, and possibly imprecise predictions. (b) In contrast, providing the model with partial graph information \widetilde{\mathcal{G}} allows us to substantially narrow down the set of causal structures that have high posterior mass p(\psi|\mathcal{D},\widetilde{\mathcal{G}}) . This yields a more concentrated posterior predictive p(y|\text{do}(t),\bm{x},\mathcal{D},\widetilde{\mathcal{G}}) , and thus more accurate predictions of the outcome. 1 Introduction Causal questions such as “what happens if a variable is intervened upon?" are central to science and decision making. Traditional approaches ( 21 ; 38 ) are tailored to specific settings where certain assumptions hold. This effectively makes estimating causal effects a two-step process: (i) make identification assumptions (e.g. through a causal graph), (ii) choose an estimator based on those assumptions. This approach has produced a sizeable toolbox of estimators, and selecting amongst them requires substantial expertise. In other areas of machine learning, a similar reliance on human expertise has been reduced by using foundation models that are trained once on diverse datasets and adapt their predictions in context to the specific task at hand, eliminating the need for bespoke estimators ( 54 ; 31 ; 3 ; 5 ) . Recently, Prior-Data-Fitted Networks (PFNs; 19 ) have been proposed as a foundation model paradigm for tabular data, achieving state-of-the-art performance on tabular benchmarks ( 14 ) , without relying on human experience to tailor a method to a specific setting. Motivated by this, several PFN-based foundation models for causal inference have been proposed ( 42 ; 10 ; 28 ; 44 ) . These Causal Foundation Models (CFM) are trained to predict causal effects on numerous synthetic datasets generated from known causal models. With the training data specifying the prior, these models implicitly integrate over a posterior on causal graphs and functions to provide the causal effect for a test dataset ( 10 ; 42 ; 44 ) . This allows for estimating causal effects by training a CFM, either with causal assumptions reflected in the training data (by restricting the causal graph), or with minimal causal assumptions (by including all possible causal graphs), while capturing uncertainty from finite samples and structural ambiguity in both cases. However, when partial domain knowledge is available, restricting the training data (prior) to reflect the domain knowledge requires expensive retraining of CFMs. On the other hand, marginalising over all structures can yield unnecessarily conservative estimates when partial domain knowledge is available, leading to imprecise predictions by assigning weight to causal worlds that experts can rule out (see Figure 1 ). We thus identify a central shortcoming of existing CFMs: They cannot flexibly condition on prior causal information when performing causal inference at test time. In this work, we study techniques for flexibly conditioning CFMs on causal information.To allow for conditioning on causal information that might be more readily available than the exact causal graph, we focus on conditioning on causal ancestral information. Even so, access to all the ancestral information can be unrealistic in certain applications. Hence, we focus on three regimes: (a) there is ancestral knowledge about all the causal variables, (b) there is partial ancestral causal knowledge among a few of the variables (which can be represented as a partial ancestral graph), and (c) there is no causal knowledge between variables. Our proposed method allows for expressing all three cases above in a single model, allowing CFMs to condition on not only observational data in-context, but also on any amount of available domain knowledge (or lack thereof). 1 1 1 The code for our natively causal prior, model training, and our experiments is available at https://github.com/ArikReuter/Graphs4CausalFoundationModels . In particular, we address the following three questions: What type of causal information should one condition on in CFMs? We argue that partial ancestral information provides a natural, flexible, and practically useful representation of causal domain knowledge, subsuming settings with no, partial, or complete causal information. How to condition on causal information? We systematically study different ways of incorporating partial graph information into CFMs and empirically show that learnable attention biases are an effective and robust mechanism to enable the use of partial ancestral information in CFMs. Does this translate to complex data? We evaluate the proposed mechanisms for incorporating partial graph information into CFMs on a complex and realistic simulator of causal structures, and on established semi-synthetic causal benchmarks. We find that ancestry-conditioning, even when it is just partially knonw, yields substantial improvements. Figure 2: Posterior predictive samples in the two-node case where T causes Y ( T\rightarrow Y ); more specifically treatment and outcome are related via Y=0.25\cdot T+\epsilon , for \epsilon\sim\mathcal{N}(0,0.1) . Here, the causal direction is not identifiable from observational data. Left : Without telling the model Q_{\theta} that is trained on arbitrary causal structures (as, e.g., in 42 or 10 ) the correct causal direction, it outputs a mixture distribution between the posteriors for the two causal directions Q_{\theta}(Y|\text{do}(T))=0.5\cdot P(Y|\text{do}(T),Y\rightarrow T)+0.5\cdot P(Y|\text{do}(T),T\rightarrow Y) . Right : When conditioned on the right causal direction, however, the model’s output aligns with the correct causal effect: Q_{\theta}(Y|\text{do}(T),T\rightarrow Y)=P(Y|\text{do}(T),T\rightarrow Y) , leading to more precise predicitons. 2 Background Our aim in this paper is the incorporation of domain knowledge into Causal Foundation models with the goal of improving their predictions of causal effects. We review the necessary background regarding causal Structures, causal interventions and interventional distributions, and prior-fitting for causal effect estimation below. 2.1 Causal Structures Causality can be formalised via structural causal models ( 38 ; 40 , SCMs;) . They consist of three main components: (i) a directed acyclic graph (DAG) \mathcal{G}=(\mathcal{N},\mathbf{A}) , where the set of nodes \mathcal{N}=\{z_{1},\ldots,z_{K}\} corresponds to random variables and the structure of the graph is encoded by an adjacency matrix \mathbf{A}\in\{0,1\}^{K\times K} . An entry A_{ij}=1 indicates the presence of a directed edge z_{i}\to z_{j} , meaning that z_{i} is a direct cause of z_{j} , while A_{ij}=0 indicates the absence of a direct causal influence. The adjacency matrix thus determines the parent set of each variable as \mathrm{pa}(z_{k})=\{z_{j}\mid A_{jk}=1\} . (ii) SCMs pose a collection of causal mechanisms \{f_{k}\}_{k=1}^{K} , where each function f_{k} determines the value of z_{k} as a function of its parents \mathrm{pa}(z_{k}) and an exogenous noise variable \varepsilon_{k} , i.e. z_{k}=f_{k}(\mathrm{pa}(z_{k}),\varepsilon_{k}), and (iii) a joint distribution p(\varepsilon_{1},\ldots,\varepsilon_{K}) over the exogenous noise variables, which are we assume to be mutually independent. We denote an SCM as the tuple \psi:=(\mathcal{G},\{f_{k}\}_{k=1}^{K},p(\varepsilon_{1},\ldots,\varepsilon_{k})) and assume no hidden variables throughout this work. SCMs can act as generative models via ancestral sampling; i.e. first drawing the exogenous variables, followed by computing the functional mechanisms of the SCM in a topological order of the nodes. This defines an observational distribution p(\mathcal{D}\mid\psi) , where \mathcal{D}=\{(\bm{x}^{(i)}_{tr},y_{tr}^{(i)})\}_{i=1}^{N_{tr}} comprises N_{tr} i.i.d samples obtained via ancestral sampling, where one (randomly determined) node z_{i} represents the target y_{tr} and the other nodes the feature-vector {\bm{x}_{tr}} . Later, we will use ancestral sampling to generate synthetic data. 2.2 Interventions and Interventional Distributions SCMs define not only the observational distribution but also distributions under interventions , formalised by the do-operator \text{do}(\cdot) . An intervention \text{do}(t) replaces the structural assignment of variable t 2 2 2 We utilise a common abuse of notation by using t to denote both the variable representing t and its value, with a constant, thereby removing all incoming edges into t in the underlying DAG. We denote an SCM that has been intervened upon with respect to variable t by \psi_{\text{do}(t)} . This operation induces the conditional interventional distribution (CID; 43 ; 47 ) p({y}\mid\bm{x},\psi_{\text{do}(t)}) , which describes, for a fixed SCM \psi , the distribution of an outcome {y} under an intervention on t , conditional on observed features \bm{x} . Access to the SCM \psi enables the computation of interventional queries with observational data ( 38 ) . In the absence of \psi , estimating the SCM from observational data is typically ill-posed: multiple SCMs can imply the same observational distribution, and point identifiability requires restrictive functional assumptions ( 40 ) . These hard restrictions on functions can lead to poor performance if they do not hold. In contrast, the Bayesian approach allows for replacing the strict functional constraints with soft constraints through a prior p(\psi) , with the posterior p(\psi|\mathcal{D}) capturing the uncertainty due to the lack of identifiability. It has been shown that under the relatively weak assumption that the prior over functional mechanisms factorises 3 3 3 This is an instantiation of the independent causal mechanism principle ( 49 ; 22 ) , the posterior over causal structure cannot be made non-informative ( 11 ) . This allows the posterior to distinguish causal direction, albeit with some theoretical probability of error ( 11 ) , and offers better performance compared with making restrictive functional assumptions for point identifiability ( 12 ; 9 ) . We incorporate factorised priors throughout our work. Any uncertainty in the posterior can be propagated to the interventional distribution by averaging over the posterior over SCMs ( 29 ) , p({y}|\text{do}(t),\bm{x},\mathcal{D})=\int p(y|\bm{x},\psi_{\text{do}(t)})p(\psi|\mathcal{D})d\psi. (1) We refer the posterior predictive in Equation 1 as the CID throughout the paper. However, computing Equation 1 is difficult due to the intractable integral, especially for complex priors p(\psi) and therefore requires approximations. 2.3 Prior-Fitting for Causal Effect Estimation Neural processes are models that meta-learn to map from datasets to a complex, usually intractable, posterior predictive distributions of interest ( 15 ; 16 ; 35 ) . Here, the training datasets form the prior distribution and the approximation of the posterior predictive side-steps any intermediate quantities. Prior-data fitted Networks (PFNs; 32 ) are a class of neural processes that learn solely from synthetic datasets sampled from a carefully constructed prior. PFNs thus are well suited for approximating Equation 1 : sampling interventional data from a distribution over SCMs is simple, and the approximation directly estimates Equation 1 , without explicitly working with the potentially complex posterior over SCMs. Training data is sampled by 1) sampling an SCM from the prior \psi\sim p(\psi) , followed by 2) sampling an observational dataset \mathcal{D}\sim p(\mathcal{D}|\psi) from the observational distribution, and 3) sampling interventional test-points from the SCM (\bm{x},{y})\sim p(\bm{x},{y}\mid\psi_{\text{do}(t)}) by drawing a value for the intervention t\sim p(t\mid\psi) , followed by propagating it through the intervened upon SCM \psi_{do(t)} together with newly sampled noise. This sampling procedure defines a joint distribution over observational-interventional data p(\mathcal{D},t,\bm{x},{y}) . CFMs can be trained in the PFN-paradigm by minimising the negative log-likelihood on data sampled from the prior. This corresponds to minimising the forward KL-divergence between the model’s approximation q_{\bm{\theta}}({y}|\text{do}(t),\bm{x},\mathcal{D}) and the CID from Equation 1 : \underset{\theta}{\text{argmin}}\ \mathbb{E}_{p(\mathcal{D},t,\bm{x},{y})}\left[-\log q_{\bm{\theta}}({y}|\text{do}(t),\bm{x},\mathcal{D})\right]=\ \underset{\theta}{\text{argmin}}\ \mathbb{E}_{p(\mathcal{D},t,\bm{x})}\left[[\mathbb{KL}\left[p({y}|\text{do}(t),\bm{x},\mathcal{D})||q_{\bm{\theta}}({y}|\text{do}(t),\bm{x},\mathcal{D})\right]\right] (2) We refer to 42 and 10 for a detailed explanation of PFNs and neural processes for causal effect estimation. 3 Related Work To the best of our knowledge, all existing Causal Foundation Model (CFM) approaches operate in the PFN-paradigm, with a distinction into two types of methods: 1) models that are only trained on data from SCMs where the interventional query is identifiable from data—a separate model is required for each identifiable case, reflecting different causal assumptions, 2) models that do not impose any assumptions and rely entirely on the posterior over SCMs. 28 propose using separate priors p_{1}(\psi),p_{2}(\psi),\ldots,p_{N}(\psi) for different causal settings (back-door adjustment, front-door adjustment, and instrumental variables). A different model q_{\theta_{1}},q_{\theta_{2}},,\ldots q_{\theta_{N}}, is trained on data from each individual prior. This ensures by a restriction of the possible graph-types per prior, that under certain assumptions a “ground-truth" causal effect can be recovered in the limit of infinite data. Furthermore, CausalPFN ( 2 ) is a PFN trained for estimating causal effects only in the back-door case, demonstrating strong performance on benchmarks that make that assumption. The above models presuppose access to causal information that renders the interventional distribution point-identifiable. In the absence of this, these models cannot be used to estimate interventional distributions. In contrast, methods like DoPFN ( 42 ) , MACE-TNP ( 10 ) , and ACTIVA 44 do not impose restrictions on the distribution over causal graphs and train on a single prior p(\psi) . These methods estimate the uncertainty due to non-identifiability, and also take advantage of the fact that the Bayesian posterior can be informative about the SCM in these cases ( 12 ; 10 ) . However, estimates from these models can still include uncertainty from SCMs that can easily be rejected through domain knowledge. As exemplified in Figure 2 , when domain knowledge is available, these models are overly conservative with their estimates, because they omit potentially critical causal information. In contrast, we argue that an all-in-one causal foundation model requires the ability to condition on both observational data and any available causal information when making predictions. A major challenge for such an approach is how to effectively condition on causal information—which is the question we tackle in this paper. 4 Methodology To address the problem of conditioning on causal information in CFMs, we aim to answer two questions: (a) What type of causal information should one condition on in CFMs? (b) How to condition on causal information in CFMs? We begin by arguing in favour of partially known ancestral matrices as an answer to question (a) : 4.1 Ancestral Relationships Beyond the direct causal connections specified via an adjacency matrix (introduced in Section 2.1 ), we can characterise the broader dependency structure of an SCM through its ancestor relationships. A variable z_{i} is said to be an ancestor of z_{j} if there exists a (possibly multi-edge) directed path z_{i}\leadsto z_{j} in the DAG. In this case, we write z_{i}\succ z_{j} to denote that z_{i} is an ancestor of z_{j} . The totality of those relationships can be represented by the ancestor matrix \mathbf{T}\in\{0,1\}^{K\times K} , defined such that T_{ij}=1 if and only if z_{i} is an ancestor of z_{j} (and 0 otherwise). Formally, \mathbf{T} corresponds to the transitive closure of the adjacency matrix \mathbf{A} , i.e. \mathbf{T}=\mathrm{sgn}(\mathbf{A}+\mathbf{A}^{2}+\mathbf{A}^{3}+\ldots) . In general, it is not possible to uniquely reconstruct an adjacency matrix from its transitive closure ( 1 ) . Ancestor matrices thus generally contain less information than adjacency matrices. This provides them with important practical advantages over working with adjacency matrices: First, ancestor matrices specify causal information, not direct causal information: Adjacency matrices require a practitioner to know for all pairs of variables (z_{i},z_{j}) if z_{i} is a direct cause of z_{j} . Ancestral matrices only require the assumption that z_{i} is a cause of z_{j} . It is, for example, uncontroversial to state that smoking causes cancer; however, assuming that smoking is a direct cause is no longer correct when closely inspecting the biological mechanisms 50 . Furthermore, one can say that ancestor matrices asymptotically contain the same causal information as adjacency matrices, under standard assumptions. This is because from an infinite observational dataset, one can determine the skeleton of the ground-truth causal DAG (Theorem 3.4 in 48 ), while checking for all pairs (i,j) if z_{i} is ancestor of z_{j} yields the remaining edge orientations. 4.2 Partial Ancestral Information While knowing ancestral instead of direct causal relationships has the meaningful practical benefits discussed above, some (or even all) of the ancestral relationships within a set of variables may be unclear. We thus propose to utilise the partially known ancestor matrix (PAM), denoted by \tilde{\mathbf{T}}\in\{-1,0,1\}^{K\times K} . Specifically, we set \tilde{T}_{ij}=1 if it is known that z_{i} is a cause of z_{j} and \tilde{T}_{ij}=-1 if it is known that z_{i} is not a cause of z_{j} . Importantly, we set \tilde{T}_{ij}=0 if the causal relationship between z_{i} and z_{j} is unknown: \tilde{T}_{ij}=\begin{cases}1,&\text{if }z_{i}\succ z_{j}\quad(z_{i}\text{ is ancestor of }z_{j}),\\ -1,&\text{if }z_{i}\not\hskip-0.7113pt\succ z_{j}\quad(z_{i}\text{ is not ancestor of }z_{j}),\\ 0,&\text{unknown }\end{cases} (3) Having partial causal knowledge that cannot be precisely specified in a causal graph is common in numerous real-world datasets. As an example, we discuss how this applies to the IHDP dataset in Appendix C . Below, we provide more general arguments for why partial ancestral information is an attractive way to specify causal assumptions: PAMs and the Arrow of Time Temporal order always provides a natural constraint on ancestral causal structure, since causes must precede effects. If variable z_{i} is measured before z_{j} in time, we immediately know that \tilde{T}_{ji}=-1 , ruling out backward causation. In practice, this type of temporal information, at least for a subset of all available variables, is arguably a very common type of causal information. PAMs and Common Causal Assumptions Considering classical causal assumptions, it straightforward to specify back-door and front-door constellations via ancestral relationships and thus PAMs. As an additional example, we discuss how unconfoundedness , an assumption commonly made in the potential outcomes framework (PO; 21 ), can be specified using PAMs in Appendix A.1 . Later, we show that encoding unconfoundedness via a PAM provides substantial benefits to the predictions of a CFM (Section 5.4 ). 4.3 Conditioning on Graphs and Consistency 28 argue in favour of priors that ensure consistency of the posterior p(y|\text{do}(t),\bm{x},\mathcal{D}) induced by the prior through constraints on the graph structure. However, using flexible priors together with jointly conditioning on observational data and graphical information \mathcal{G}_{0} (for instance via PAMs) yields consistency whenever \mathcal{G}_{0} makes the CID identifiable. This means that consistency depends on the available information \mathcal{G}_{0} , not the prior the model was trained on. This is arguably much more natural for an all-in-one CFM, since a single model is capable of (consistently) estimating causal effects while also being able to quantify uncertainty due to identifiability of causal effects. We provide a more detailed discussion using standard results from Bayesian nonparametrics in Appendix B . 4.4 Architectures for Conditioning on Graphs in Causal Foundation Models Now, we discuss architectural modifications for conditioning on graphical causal information in the form of partially known ancestral matrices, introduced in Section 4.2 . Graph-Conditioned Architecture Following 10 ; 42 , we employ a transformer-based architecture ( 52 ) . We begin by embedding the input data such that the model (i) can capture intra-sample correlations, (ii) preserves the specific value of each variable, and (iii) can distinguish between different variable roles (e.g. feature vs. treatment) and data types (observational vs. interventional). The exact embedding procedure is detailed in Appendix D . We then apply multiple transformer blocks with a factorised attention mechanism that alternates between two operations at each layer: (i) sample-wise attention , which exchanges information across different samples, and (ii) feature-wise attention , which models relationships between variables within a sample. Our core contribution lies in the feature-wise stage, where we explicitly condition the model on the partial ancestral matrix \tilde{T} . We propose two complementary mechanisms for injecting this structural information: structural attention biasing and structural feature modulation . We empirically investigate the optimal configuration in Section 5.1 . Structural Attention Biasing To guide the information flow between variables (nodes), we inject ancestry information directly into the feature-wise self-attention mechanism. Let \mathbf{A}_{ij} denote the attention logit between feature i (query) and feature j (key). We define a bias matrix \mathbf{B}\in\mathbb{R}^{K\times K} derived from \tilde{T} to modulate attention scores between a query (effect) and a key (potential cause). We explore two variants: 1.) Soft Attention Bias: We introduce learnable scalar biases that encourage attention towards known ancestors and discourage attention towards known non-ancestors. The attention logits are modified as \mathbf{A}_{ij}+\mathbf{B}_{ij} , where \mathbf{B}_{ij}=\begin{cases}+\beta_{\text{anc}}&\text{if }\tilde{T}_{ij}=1\ \ \ \quad\text{(ancestor)}\\ -\beta_{\text{non-anc}}&\text{if }\tilde{T}_{ij}=-1\quad\text{(non-ancestor)}\\ 0&\text{if }\tilde{T}_{ij}=0\ \ \ \quad\text{(unknown)}.\end{cases} (4) Here, \beta_{\text{anc}} and \beta_{\text{non-anc}} are learnable parameters. This soft formulation allows the model to prioritise causal parents while remaining robust to incomplete graph information. 2.) Hard Masking: Alternatively, we enforce sparsity by setting \mathbf{B}_{ij}=-\infty if \tilde{T}_{ij}=-1 , and \mathbf{B}_{ij}=0 otherwise. This strictly prohibits information flow from variables known not to be causes while treating ancestor and unknown node relationships equally. Structural Modulation (GCN) Complementary to local attention biasing, we condition the model on a global representation of the ancestry matrix \tilde{T} using a Graph Convolutional Network (GCN; 13 ). We initialise node features using learnable role embeddings (feature, treatment, and outcome) and process them to obtain structure-aware node embeddings \mathbf{Z}_{\text{graph}} . These embeddings are injected via Adaptive Layer Normalisation (AdaLN; 39 ). In each block, scale and shift parameters are derived from \mathbf{Z}_{\text{graph}} , modulating feature statistics based on each node’s position in the causal graph. Sample-wise Attention and Decoding After graph-conditioned feature processing, the model performs sample-wise attention to aggregate information across the dataset. We implement this using Multi-Head Self-Attention over the observational data, followed by Multi-Head Cross-Attention from the query to the observational set. This process is repeated over L blocks. Finally, we extract the outcome representation for the query sample and project it through a linear head to parameterise a discretised bar distribution ( 19 ) over the outcomes. Complete implementation details are provided in Appendix D . Figure 3: Performance comparison of different graph-conditioning methods on linear-Gaussian data. Results are shown as the improvement in in negative log-likelihood (NLL), in terms of mean-squared-error (MSE) and coefficient of determination ( R^{2} ) on held-out test data relative to a non-graph-conditioned baseline. The error-bars represent 95 -percent bootstrap confidence intervals for the median. We compare soft and hard attention biases as well as a GCN-based conditioning approach, using either ground-truth adjacency matrices (Adj.) or ancestor matrices (Anc.) as graph input. All methods yield significant benefits with Soft Attention based methods performing the best. 5 Experiments We empirically study the effectiveness of graph-conditioning in Causal Foundation Models (CFMs), progressing from controlled synthetic settings to complex and semi-synthetic benchmarks. 5.1 Which Graph-conditioning Method Works Best? Our primary objective is to enable Causal Foundation Models (CFMs) to condition effectively on ancestral knowledge. As outlined in Section 4.4 , this can be achieved through several distinct mechanisms, and our first research question seeks to identify the most effective approach. To isolate the contribution of the graph information, we first benchmark these methods on synthetic case studies where the graph information is critical because the underlying causal graph is not identifiable from the observational data alone. To achieve this, we train a model on 30,000 synthetic datasets sampled from SCMs with up to 50 nodes, linear mechanisms, internal standardisation, and Gaussian noise (see Appendix F.1 ). Then we evaluate on a test-set sampled from the same distribution. Figure 3 presents the improvement in Negative Log-Likelihood (NLL), Mean-Squared Error (MSE) and the R^{2} score relative to a baseline that is trained without graph-conditioning but otherwise identical. We also evaluate two types of conditioning information: the ground-truth adjacency matrix (Adj.) and the ancestor matrix (Anc.), with the latter providing in principle less information while often being a more realistic assumption in real-world scenarios, as argued in Section 4.1 . As shown in Figure 3 , all graph-conditioned variants substantially outperform the baseline, confirming that the model effectively leverages structural information. Among the investigated architectures, Soft Attention and the combination of Soft Attention with a GCN consistently outperforms both Hard Attention and the GCN-only variants. When comparing conditioning signals (Adj. vs. Anc.), we generally only observe a small divergence in performance, empirically confirming that CFMs can be effectively conditioned on ancestor information. Interestingly, using the ancestor matrix instead of the adjacency matrix yields small benefits in terms of MSE and R^{2} , especially for the soft-attention-based methods. We hypothesise that restricting attention solely to direct parents (as implied by the adjacency matrix) may be overly limiting for bias-based mechanisms, which benefit from the broader context provided by the ancestor matrix. Detailed results can be found in Section H.1 . 5.2 Is Graph-Conditioning still Effective with Partial, rather than Full Ancestry Information? In order to support no, full and partial conditioning, we need to amortise during training over scenarios with varying amounts of available ancestor information. This raises the question of how much performance the model loses due to the resulting challenges this entails. To obtain an answer, we compare a model capable of handling any amount of missing information (amortised) with specialised models: one specialised in never using ancestral information; one specialised in always using full ancestral information. As observed in Figure 9 in Appendix H.2 , the amortised procedure during training is efficient, with the performance of the amortised model being comparable to that of the specialised models. This shows that our training procedure yields a model that is flexible concerning the amount of information it receives without sacrificing performance—a single model can effectively handle varying amounts of available information. Figure 4: Performance on complex synthetic data for different fractions of hidden entries in the ancestor matrix for the Soft Attention (Ancestor) and GCN + Soft Attention (Ancestor) graph-conditioning approaches. Results are measured as the absolute difference to a baseline that does not support graph conditioning but is otherwise trained identically. The error-bars represent 95-percent bootstrap confidence intervals for the mean. Providing full graph information significantly improves performance while GCN + Soft Attention performs better than just using Soft Attention. 5.3 Does Graph-conditioning Help with Complex Causal Structures? Besides the simple linear-Gaussian mechanisms that we considered in the previous sections, we now turn to significantly more complex synthetic data to evaluate the effect of graph-conditioning, representing a more realistic test-bed. To obtain such a challenging and realistic simulator for this evaluation, we developed a custom prior over SCMs that we use to train and evaluate different models: An Empirically Validated Causal Prior To the best of our knowledge, all existing priors for tabular foundation models that perform well on predictive tasks are not natively causal and do not directly support the generation of interventional data ( 19 ; 41 ; 56 ) . To evaluate our graph-conditioning procedure on representative data, we therefore developed a custom, natively causal prior based on the relatively simple priors proposed by 42 ; 10 and developments proposed by 41 . We aim to use not only a complex but also a realistic simulator as a test-bed for the graph-conditioning mechanisms—a prior that is similar to what one can expect CFMs (including future CFMs) to be trained on. In order to confirm that this prior yields realistic observational data, we train a tabular foundation model in the purely predictive setup and compare its performance on small-scale tabular regression datasets, where we observe accurate predictions compared to common baselines. See Appendix E.1 for details and a discussion about existing priors. We evaluate the two best graph-conditioning mechanisms from section F.1 , namely Soft Attention and Soft Attention combined with a GCN on 30,000 datasets from the complex prior. We vary the fraction of entries in the ancestor matrix that are hidden, i.e., replaced by 0 , and measure the absolute improvement compared to a model that is not trained with ancestor-conditioning, but otherwise identical. First, Figure 4 shows that providing full graph-information significantly improves the error of the model in all metrics and for both graph-conditioning methods. When we increase the fraction of hidden entries in the ancestor matrix towards one, the models’ performance degrades and is, for MSE and R^{2} , no longer significantly different from the non-graph-conditioned baseline. The NLL metric indicates a consistent benefit of training models with ancestral information on the complex mechanisms, even when no ancestral information is provided. We hypothesise that this is due to improved training dynamics in when the causal structure of the data is respected during training. In terms of the NLL, one can observe that the combined GCN + Soft Attention approach performs noticeably better, than just using Soft Attention, while in terms of the less sensitive MSE and R^{2} metrics, which assess only the mean of the PPD, there is a systematic yet not significant difference. 5.4 Does Partially Known Ancestral Information Help on Semi-Synthetic Data? Finally, we also investigate whether providing our model with graph information offers benefits on the commonly used semi-synthetic RealCause Benchmark ( 34 ) . The data for this benchmark is simulated to satisfy the unconfoundedness assumption, which does not specify the causal relationship between the covariates. This makes it impossible to directly use complete adjacency matrices or full ancestral matrices to exactly represent this assumption; however, it can can be specified in terms of partially known ancestral matrices ( Proposition A.1 ). To evaluate the RealCause benchmark, we train a model on the complex-mechanism prior from Section 5.3 and assess the effect of providing ancestral information. Fixing the prior while using a model that (a) is trained in the predictive setup, (b) uses the ancestral information, and (c) does not use the ancestral information eliminates the confounding effect of different priors, thus allowing for a precise assessment of the effect of conditioning on ancestor information. We compare the performance of the model without any graph information (“No Ancestral Info.”) to the model with the correct graph information (“Ancestral Info.”). Furthermore, we compare against a predictive model (“Predictive”) trained on the complex-mechanisms prior, as under unconfoundedness, predicting the causal effect effectively becomes a predictive task ( 26 ) . Table 1 shows that providing the model trained on the generic complex prior with the partially known ancestral matrix systematically improves performance compared to not conditioning on ancestral information for the IHDP, ACIC and CPS datasets. For PSID, all methods achieve suboptimal performance, which we find is due to the highly imbalanced distribution of treated versus untreated individuals. When performing rebalancing, all methods substantially improve while conditioning on the ancestral information yields substantial benefits (see Appendix F.3 ). Table 1: Providing graph information helps for semi-synthetic data: We evaluate the effect of providing partially known ancestral information encoding unconfoundedness on semi-synthetic RealCause datasets. We report the performance for a predictive model trained on our complex prior, a model with partially known ancestral information , and a model with no ancestral information . As metrics, we use root Precision in Estimation of Heterogeneous Effect ( \sqrt{\text{PEHE}} ) and relative error when predicting the ATE \epsilon_{ATE} (See Section F.2.1 ). IHDP ACIC CPS PSID (unbalanced) Method \sqrt{\text{PEHE}}\ (\downarrow) \epsilon_{\text{ATE}}\ (\downarrow) \sqrt{\text{PEHE}}\ (\downarrow) \epsilon_{\text{ATE}}\ (\downarrow) \sqrt{\text{PEHE}}\ (\downarrow) \epsilon_{\text{ATE}}\ (\downarrow) \sqrt{\text{PEHE}}\ (\downarrow) \epsilon_{\text{ATE}}\ (\downarrow) Predictive 6.79 \scriptstyle\pm 0.81 0.81 \scriptstyle\pm 0.11 3.14 \scriptstyle\pm 0.47 0.38 \scriptstyle\pm 0.06 11393 \scriptstyle\pm 31 0.78 \scriptstyle\pm 0.00 11820 \scriptstyle\pm 41 1.03 \scriptstyle\pm 0.01 No Ancestral Info. 6.28 \scriptstyle\pm 0.79 0.67 \scriptstyle\pm 0.05 3.47 \scriptstyle\pm 0.47 0.46 \scriptstyle\pm 0.09 12800 \scriptstyle\pm 55 0.99 \scriptstyle\pm 0.01 13096 \scriptstyle\pm 26 0.98 \scriptstyle\pm 0.00 Ancestral Info. 5.49 \scriptstyle\pm 0.78 0.49 \scriptstyle\pm 0.08 2.79 \scriptstyle\pm 0.45 0.17 \scriptstyle\pm 0.08 11213 \scriptstyle\pm 60 0.70 \scriptstyle\pm 0.02 12975 \scriptstyle\pm 24 1.09 \scriptstyle\pm 0.01 6 Discussion In this work, we study how to condition CFMs on structured causal knowledge. We argue why partially known ancestral matrices provide a realistic and expressive way to encode domain knowledge, capturing the common case where some causal relationships can be known, some ruled out, and some uncertain. Architecturally, we identify that the most efficient method to inject this information is by combining attention biassing with a graph convolutional encoder. Using this strategy, we enable the construction of a single CFM that accounts for uncertainty when specific causal domain knowledge is lacking, while effectively leveraging varying amounts of available causal information, which we demonstrate on complex, non-identifiable synthetic data as well as semi-synthetic benchmarks. This work focusses on graphical assumptions encoded through partially known ancestry. While this is a widely applicable form of causal knowledge, CFMs could also benefit from flexibly conditioning on aspects of functional mechanisms or noise distributions. Moreover, while we evaluate on a complex causal prior, validated in the predictive setting, the development of practically useful “all-in-one” CFMs would require comprehensive real-world causal benchmarks. Although such benchmarks currently do not exist, they would not only enable the development of foundation models covering diverse real-world use cases but also provide rigorous testing grounds to optimise how domain knowledge is incorporated. Despite these current constraints, our method successfully demonstrates that conditioning on causal information is both feasible and effective. This establishes a promising foundation for the next generation of CFMs, bringing us closer to robust, all-in-one causal effect estimation models. Appendix A Detailed Propositions Proposition A.1 (Unconfoundedness from partial graph knowledge) . Assume a causally sufficient SCM on the observed variables \mathcal{N}=\{t,y,x_{1},\dots,x_{K}\} with potential outcomes y_{0},y_{1} , a binary treatment t\in\{0,1\} , and y=ty_{1}+(1-t)y_{0}. Let \tilde{\mathbf{T}} be a PAM such that: • t\leadsto y is specified in the PAM, • for all i , x_{i}\leadsto t and x_{i}\leadsto y are specified in the PAM, • no variable outside \bm{x}=\{x_{i}\}_{i=1}^{K} is allowed to be an ancestor of both t and y (i.e. the PAM excludes any common ancestor of t and y outside x ). Then the unconfoundedness condition holds: (y_{0},y_{1})\ \perp\!\!\!\perp\ t\mid x. Proof sketch. The PAM constraints enforce that all common causes of t and y are contained in \bm{x} . Since the model is causally sufficient, there are no unobserved confounders. Conditioning on \bm{x} therefore removes all shared sources of variation between the treatment assignment mechanism and the outcome mechanism. Intervening on t changes only the structural equation of t and does not introduce new dependence between t and (y_{0},y_{1}) once \bm{x} is fixed. Hence, conditional on \bm{x} , treatment assignment is independent of the potential outcomes, which is exactly the unconfoundedness assumption (y_{0},y_{1})\ \perp\!\!\!\perp\ t\mid\bm{x} . ∎ Appendix B A Discussion on Consistency through Graphical Information Suppose a ground-truth DAG \mathcal{G}^{*} with ground-truth mechanisms-and distribution-parameters \theta^{*} fully specifying an SCM \psi^{*} . Assume we have an observational dataset \mathcal{D}_{n} of size n sampled from P(\mathcal{D}_{n}|\psi^{*}) . Also, assume that we have correct, but potentially partial graph information \mathcal{G}_{0} about \mathcal{G}^{*} , which, together with the parameter vector \theta^{*} , uniquely identifies the CID P(y|\text{do}(t),\bm{x},\theta^{*},\mathcal{G}_{0}) , such that P(y|\text{do}(t),\bm{x},\theta^{*},\mathcal{G}_{0})\neq P(y|\text{do}(t),\bm{x},\theta^{\prime},\mathcal{G}^{\prime}) for any tuple (\theta^{\prime},\mathcal{G}^{\prime})\neq(\theta^{*},\mathcal{G}_{0}) . The question we are interested in is whether, for infinitely many observational data points n\to\infty , the CID P(Y|\text{do}(t),\bm{x},\mathcal{D}_{n},\mathcal{G}_{0}) will converge to identifiable CID P(Y|\text{do}(t),\bm{x},\theta^{*},\mathcal{G}_{0}) . To adress this question, assume a parametrisation such that \theta\mapsto P(\mathcal{D}_{n}|\theta,\mathcal{G}_{0}) is injective, which might require working in function-space. Consistency arguments from Bayesian nonparametrics yield (under their respective conditions): P(\theta|\mathcal{G}_{0},\mathcal{D}_{n})\rightarrow\delta(\theta^{*}). One can use Doob’s theorem for this type of consistency; which requires \theta to be finite-dimensional, and ensures consistency almost surely in terms of P(\theta) , but otherwise only needs measurability of the map \theta\mapsto P(\mathcal{D}_{n}|\theta,\mathcal{G}_{0}) . See 30 (Theorem 2.4). One can also make sure of more more powerful results such as Theorem 3 by 8 , or Corollary 1 by 53 , also used by 33 in the context of PFNs. Now, under regularity conditions on the functional P(\theta|\mathcal{G}_{0},\mathcal{D}_{n})\mapsto\int P(y|do(t),\bm{x},\theta,\mathcal{G}_{0})P(\theta|\mathcal{G}_{0},\mathcal{D}_{n})d\theta , this implies: P(Y|\text{do}(t),\bm{x},\mathcal{D}_{n},\mathcal{G}_{0})\to P(Y|\text{do}(t),\bm{x},\theta^{*},\mathcal{G}_{0}), which is exactly consistency to the identifiable CID. If the partial graphical information \mathcal{G}_{0} , together with \theta^{*} fully identifies the SCM \psi^{*} , one has P(Y|\text{do}(t),\bm{x},\theta^{*},\mathcal{G}_{0})=P(Y|\text{do}(t),\bm{x},\psi^{*}) , i.e. consistency in terms of the ground-truth SCM \psi^{*} . One such regularity condition is continuity of P(\theta|\mathcal{G}_{0},\mathcal{D}_{n})\mapsto\int P(y|do(t),\bm{x},\theta,\mathcal{G}_{0})P(\theta|\mathcal{G}_{0},\mathcal{D}_{n})d\theta , which allows to use the Continuous Mapping Theorem (Theorem 2.3 in 51 ). Alternatively, it suffices that P(y|do(t),\bm{x},\theta,\mathcal{G}_{0})P(\theta|\mathcal{G}_{0},\mathcal{D}_{n}) is a bounded and continuous function of \theta by the Portmanteau Theorem (Lemma 2.2 in 51 ). Appendix C Example for Partial Ancestral Information One of the most frequently used datasets to benchmark causal effect estimation is the the Infant Health and Development Program (IHDP) dataset ( 4 ) . Here, a partially known causal structure arises naturally from temporal, biological, and logical considerations. Maternal characteristics such as age, race, and endocrine or nervous system disorders clearly precede and can plausibly influence prenatal behaviors like cigarette consumption, implying \tilde{T}_{\text{maternal\_age,cigarettes}}=1 . Likewise, these maternal and prenatal factors are known to affect birth-related outcomes such as the child’s bilirubin level ( \tilde{T}_{\text{cigarettes,bilirubin}}=1 ), whereas reverse effects such as \tilde{T}_{\text{bilirubin,cigarettes}}=-1 can be ruled out. Social and demographic variables such as maternal race or birthplace are fixed attributes and thus cannot be caused by medical conditions or behaviours ( \tilde{T}_{\text{race,medical}}=1 , \tilde{T}_{\text{medical,race}}=-1 ). In contrast, many dependencies among medical covariates, such as between endocrine and nervous system conditions, remain uncertain. Appendix D Details on the Architecture Our architecture shares a lot of similarities with the architectures proposed in 42 ; 10 , while also allowing for conditioning on partial ancestral information. We assume we observe N_{tr} samples from the observational data \mathcal{D}=\{{\bm{x}^{(i)}_{tr}},t^{(i)},y_{tr}^{(i)})\}_{i=1}^{N_{tr}} and we aim to predict the distribution of the interventional outcome {y} given a set of interventional features \bm{x} and a treatment value t : p({y}\mid\text{do}(t),\bm{x},\mathcal{D}) . We parameterise the latter with a bar distribution following 42 ; 19 . Data Pre-Processing The input to the model is comprised of 1) the observational data \mathcal{D}=\{{\bm{x}^{(i)}_{tr}},t^{(i)},y_{tr}^{(i)})\}_{i=1}^{N_{tr}}\in\mathbb{R}^{N_{tr}\times(D+2)} , where D represents the maximum number of features (excluding the treatment and outcome) used during training, and 2) the interventional query ( \bm{x}\in\mathbb{R}^{D} , t\in\mathbb{R} ). If the dataset of interest contains fewer than D features we pad the input with zeros. Embeddings Before applying any attention mechanism, we aim to embed the data such that the model 1) is able to capture intra-sample correlations amongst features, 2) can preserve the specific value of each variable, and 3) is able of distinguishing between the different types of variables (i.e. feature vs. treatment) and data sources (observational vs. interventional). Let \mathbf{X}\in\mathbb{R}^{S\times D} and \mathbf{t}\in\mathbb{R}^{S\times 1} denote a batch of S samples (where S=N_{tr}+1 ) comprising both observational data and the interventional query. We first concatenate features and treatments to form the input tensor \mathbf{V}=[\mathbf{X};\mathbf{t}]\in\mathbb{R}^{S\times(D+1)} . This input is processed to produce embeddings in \mathbb{R}^{S\times(D+1)\times d_{\text{model}}} , where d_{\text{model}} is the hidden dimension of the model: 1. Row Embeddings: To capture intra-sample correlations, we process the entire feature vector of a sample simultaneously. The concatenated input \mathbf{V} is projected via a Multi-Layer Perceptron ( \text{MLP}_{\text{row}}:\mathbb{R}\to\mathbb{R}^{d_{\text{model}}} ) to generate a global sample embedding. This embedding is then broadcast across the feature dimension to yield \mathbf{E}_{\text{row}}\in\mathbb{R}^{S\times(D+1)\times d_{\text{model}}} . 2. Cell Embeddings: To preserve the precise value of individual variables, we apply a separate MLP ( \text{MLP}_{\text{cell}}:\mathbb{R}^{1}\to\mathbb{R}^{d_{\text{model}}} ) element-wise to each scalar entry in \mathbf{V} . To distinguish between variable types (feature vs. treatment) and data sources (observational vs. interventional), we add learnable role embeddings \mathbf{e}_{\text{role}}\in\mathbb{R}^{d_{\text{model}}} to the output: \mathbf{E}_{\text{cell}}^{(i,j)}=\text{MLP}_{\text{cell}}(\mathbf{V}_{ij})+\mathbf{e}_{\text{type}}[j]+\mathbf{e}_{\text{source}}[i] (5) where \mathbf{e}_{\text{type}} distinguishes between features ( 1\dots D ) and the treatment column ( D+1 ), and \mathbf{e}_{\text{source}} distinguishes between observational samples ( 1\dots N_{tr} ) and the query sample. The final embedding \mathbf{H}^{(0)}\in\mathbb{R}^{S\times(D+1)\times d_{\text{model}}} is a learnable weighted sum of these two representations: \mathbf{H}^{(0)}=\lambda_{\text{row}}\cdot\mathbf{E}_{\text{row}}+\lambda_{\text{cell}}\cdot\mathbf{E}_{\text{cell}} (6) where \lambda_{\text{row}} and \lambda_{\text{cell}} are learnable scalar parameters initialized to control the relative importance of global context versus local values. Finally, we append a learnable token representing the target variable y , resulting in a final input sequence shape of S\times(D+2)\times d_{\text{model}} . Feature Positional Encodings Inspired by 41 , we additionally inject feature positional encodings. Although the architecture already breaks the symmetry through the use of row embeddings, we further assure that we avoid representational collapse through sinusoidal positional encodings. For the sequence of D+2 tokens (representing the D features, the treatment indicator, and the target variable), we compute the encoding \mathbf{P}\in\mathbb{R}^{(D+2)\times d_{\text{model}}} as: \mathbf{P}{(pos,2i)}=\sin\left(\frac{pos}{10000^{2i/d{\text{model}}}}\right),\quad\mathbf{P}{(pos,2i+1)}=\cos\left(\frac{pos}{10000^{2i/d{\text{model}}}}\right) (7) These fixed encodings are broadcast across the batch and sample dimensions and added element-wise to the variable embeddings \mathbf{H}^{(0)} . Attention Sinks Following 17 , we prepend a set of N_{\text{sink}} learnable dummy samples to the input sequence in order to improve optimisation stability and prevent the attention heads from collapsing when no relevant context is found. We instantiate these sinks as learnable parameters \mathbf{S}_{\text{feat}}\in\mathbb{R}^{N_{\text{sink}}\times(D+1)\times d_{\text{model}}} (corresponding to features and treatment) and \mathbf{S}_{\text{target}}\in\mathbb{R}^{N_{\text{sink}}\times 1\times d_{\text{model}}} (corresponding to the target variable). These are concatenated to form complete dummy samples \mathbf{S}=[\mathbf{S}_{\text{feat}};\mathbf{S}_{\text{target}}] and prepended to the sample axis of the input batch. After performing the above-mentioned modifications, the input to the transformer backbone \tilde{\mathbf{H}}^{(0)} has dimensions (N_{tr}+1+N_{\text{sink}})\times(D+2)\times d_{\text{model}} . Graph Conditioning Mechanism The main contribution of our work is enabling a causal foundation model to condition on partial ancestry information when this is available. We explore several ways in which this conditioning can be performed: 1) through structural biasing in the attention mechanism with the use of masks, or 2) by constructing structural embeddings via a Graph Convolutional Network (GCN) ( 13 ) and injecting the information into each layer of the transformer through conditional layer-normalisation ( 39 ) . Structural Biasing (Attention Mask) We utilise the partial ancestry matrix \tilde{T}\in\{-1,0,1\}^{(D+2)\times(D+2)} to encourage the attention mechanism to respect the underlying causal structure encoded in the matrix. We define a bias matrix \mathbf{B} based on the relationship between a query node i (the effect) and a key node j (the potential cause) in two modes: • Soft Attention Bias: We introduce a learnable bias to encourage attention towards known ancestors and discourage it towards non-ancestors. The attention scores are modified by adding a bias term \mathbf{B}_{ij} : \mathbf{B}_{ij}=\begin{cases}+\beta_{\text{anc}}&\text{if }\tilde{T}_{ij}=1\text{ (Known Ancestor)}\\ -\beta_{\text{non-anc}}&\text{if }\tilde{T}_{ij}=-1\text{ (Known Non-Ancestor)}\\ 0&\text{if }\tilde{T}_{ij}=0\text{ (Unknown)}\end{cases} (8) where \beta_{\text{anc}} and \beta_{\text{non-anc}} are learnable positive scalars. This soft bias encourages the model to respect the causal relationships encoded in the matrix, while retaining the flexibility to attend to other variables and thus robustness to incomplete graph information. • Hard Attention Mask (Ablation Baseline): We also study a variant where we enforce strict sparsity. We set \mathbf{B}_{ij}=-\infty if \tilde{T}_{ij}=-1 (masking known non-ancestors), and \mathbf{B}_{ij}=0 otherwise. This strictly prevents variables from attending to nodes known not to be their causes. In both cases, we implicitly set the diagonal \tilde{T}_{ii}=1 to ensure every variable always attends to itself. Graph Encoding (GCN) We begin by constructing a graph representation where nodes correspond to the variables in the input sequence. Given the partial ancestry matrix \tilde{T}\in\{-1,0,1\}^{(D+2)\times(D+2)} , we first initialise node features \mathbf{Z}^{(0)}\in\mathbb{R}^{(D+2)\times d_{\text{model}}} using learnable role embeddings specific to each node type (feature, treatment, or outcome). These initial features are processed by a Graph Convolutional Network (GCN) encoder operating on \tilde{T} to produce structure-aware node embeddings \mathbf{Z}_{\text{graph}} . These embeddings subsequently modulate each transformer block via Adaptive Layer Normalisation (AdaLN) ( 39 ) . Transformer Architecture We follow 10 and use a transformer architecture ( 52 ) with factorised attention that alternates between processing interactions between features (intra-sample) and between samples (inter-samples). The input to the transformer are the obtained embeddings \tilde{\mathbf{H}}^{(0)}\in\mathbb{R}^{(S+N_{\text{sink}})\times(D+2)\times d_{\text{model}}} , as well as the attention biases \mathbf{B} and the graph encodings \mathbf{Z}_{\text{graph}} if the GCN mechanism is used. 1. Feature-wise Attention (Graph-Conditioned) This layer operates independently on each sample to capture the dependencies between variables. We employ a standard Multi-Head Self-Attention (MHSA) mechanism augmented with the graph conditioning strategies described previously, such that the output of layer l becomes: \mathbf{H}^{\prime(l)}=\text{MHSA}\left(\text{AdaLN}(\mathbf{H}^{(l-1)},\mathbf{Z}_{\text{graph}}),\text{mask}=\mathbf{B}\right)+\mathbf{X} (9) where we introduce the conditioning on the GCN embeddings \mathbf{Z}_{graph} if they are available, and \mathbf{B} applies the structural soft bias. If no GCN embeddings are provided AdaLN is replaced with a LayerNorm. 2. Sample-wise Attention (Inductive) This layer operates independently on each feature dimension to exchange information across samples. Similarly to 10 , we first perform MHSA amongst the observational data, followed by Multi-Head Cross Attention (MHCA) from the queries to the observational data for a reduced computational cost (as opposed to masked MHSA). 3. Position-wise Feed-Forward Network The final sub-layer is a standard two-layer MLP with SwiGLU activation ( 46 ) , applied to every token independently. Similar to the feature attention layer, this MLP is conditioned via AdaLN if the graph embeddings are provided, otherwise we use LayerNorm: \mathbf{H}^{(l)}=\text{MLP}(\text{AdaLN}(\mathbf{H}^{\prime(l)},\mathbf{Z}_{graph}))+\mathbf{H}^{\prime(l)} (10) Decoding After processing through L layers, we obtain the final latent representation \mathbf{H}^{(L)}\in\mathbb{R}^{(S+N_{\text{sink}})\times(D+2)\times d_{\text{model}}} . To produce a prediction for the interventional query, we slice this tensor to extract the representation corresponding to the outcome node of the interventional sample, denoted by \mathbf{h}_{\text{intv}}\in\mathbb{R}^{d_{\text{model}}} . Finally, we project \mathbf{h}_{\text{intv}} through a linear prediction head ( \text{Linear}:\mathbb{R}^{d_{\text{model}}}\rightarrow\mathbb{R}^{K} ) to obtain the parameters of the interventional outcome distribution. Bar Distribution Following 19 , we parameterise the interventional outcome distribution using the bar distribution. This approach partitions the outcome space into K bins with supports (a_{0},a_{1}],(a_{1},a_{2}],\ldots,(a_{K-1},a_{K}] . We fix K=1000 during training and use equally sized bins. Standardising the outcome variable y both during training and inference ensures a bounded co-domain. For each interventional query, the model outputs a vector of logits \bm{\lambda}\in\mathbb{R}^{K} . Applying a softmax transformation yields a categorical distribution over bins, \bm{\Delta}=\mathrm{softmax}(\bm{\lambda}) . The resulting bar distribution is given by q_{\theta}(y\mid\bm{\lambda})=\sum_{k=0}^{K-1}\mathbb{I}(a_{k}