DAG-VAERL: a novel causal inference method for building causal gene regulatory networks Authors: Teng Long, Sachit Satyal, Yong-Fang Kuo, Jean Gao Published: 2026 ABSTRACT BackgroundCausal discovery methods provide a powerful tool for uncovering the causal relationships between lncRNAs and target genes in gene regulations. In many causal inference and structure learning tasks, learning the Directed Acyclic Graph (DAG) structure from data is a challenging problem. Traditional DAG learning methods often rely on heuristic searches or strict constraints, which fail to effectively handle complex nonlinear relationships and discrete data.MethodsTo address this, we propose a novel deep generative model—DAG-VAERL, which combines Graph Neural Networks (GNN) as well as Reinforcement Learning (RL) frameworks and Graph Attention Networks (GAT) module, leveraging Variational Autoencoders (VAE) to learn the DAG structure. DAG-VAERL is capable of modeling complex dependencies between nodes through GNNs and optimizing the graph structure using RL strategies.ResultsWe conduct extensive experiments on synthetic and real-world datasets, including Alzheimer’s disease data, to validate the superiority of DAG-VAERL in structural discovery and parameter estimation. Experimental results demonstrate that DAG-VAERL significantly outperforms traditional methods in structure recovery, especially when dealing with complex data involving nonlinear and discrete variables.ConclusionsThis model not only effectively learns the DAG structure from data but also serves as a powerful tool for causal inference and other graph-bard analysis tasks, providing a new approach for related fields. Introduction Long non-coding RNAs (lncRNAs) play a critical role in gene regulation, and their dysregulation is closely associated with the development of various diseases. Causal discovery methods provide a powerful tool for uncovering the regulatory relationships between lncRNAs and target genes. By identifying causal relationships between lncRNAs and genes, we can gain deeper insights into how lncRNAs participate in gene regulatory networks and elucidate their specific mechanisms in cellular processes. For instance, certain lncRNAs can directly or indirectly regulate the expression of downstream genes by interacting with transcription factors or chromatin-modifying complexes. The identification of such causal relationships not only helps clarify the functions of lncRNAs but also offers potential molecular targets for disease diagnosis and treatment. Therefore, causal discovery is of great significance in lncRNA research, advancing our comprehensive understanding of gene regulatory networks. Causality has been widely used in various fields, such as weather forecasting, economics, medical imaging, and medical treatment. The fundamental of causal inference is learning of directed acyclic graph (DAG), which could be considered as one form of Bayesian Network. To gain the relationship of causality, many experiments are necessary that are not limited to the use of interventions or manipulations. However, such experiments are not possible due to many factors, including expensive cost, moral reasons, and inapplicability [1]. The various methods of learning causality have been developed and each method has its advantages. Basically, three methods of causal discovery are considered as widely-used, constraint-based, score-based, and continuous-optimization-based methods. The traditional way (constraint-based) of finding causality is NP-hard. PC algorithm [2] is the most famous example of constraint-based methods. This kind of method requires tests for conditional independencies to build the skeleton of the graph, and then, d-separation is used to discover the directions of most edges. This often results in a partially directed acyclic graph (PDAG) , as not all edge directions can be uniquely determined. However, excessive conditional independencies cause high complexity. The score-based method uses some scoring function to calculate the score of the candidate graphs. Those scoring functions include the Bayesian Information Criterion (BIC) [3], the Minimum Description Length [4], the Bayesian Gaussian equivalent (BGe)score [3] and others. But all the scoring functions require complex space, especially when the number of nodes is numerous, which leads to challenges for optimization. To address these scalability and efficiency bottlenecks inherent in traditional combinatorial graph search, several advanced structure learning strategies have been recently proposed. For instance, PSL [5] introduces an efficient algorithm tailored for partial Bayesian network structure learning, significantly reducing the search space. Similarly, F2SL [6] leverages feature selection mechanisms to enable a highly scalable local-to-global structure discovery process. While these discrete search algorithms have made substantial progress in improving computational efficiency, they still face challenges when capturing complex, non-linear causal dynamics in high-dimensional continuous spaces. Currently, thanks to the rise of deep learning, continuous-optimization-based methods combines causal discovery and optimization by deep learning. NOTEARS [7] introduced an equivalent acyclicity constraint that changed the old way of testing conditional independencies to continuously optimizing based on the constrained function. The constraint function is the key to ensuring that the graph built by the adjacency matrix is a directed acyclic graph. DAG-GNN [8] extends the NOTEARS [7] constraint function by combining it with VAE. They used KL divergence to calculate the loss of the original distribution and the predicted distribution and continuously optimize it by the augmented Lagrangian approach. Reinforcement learning has also been introduced into DAG structure learning to improve the exploration of the combinatorial graph search space. Zhu et al. [9] proposed an RL-based causal discovery framework in which an encoder-decoder policy network generates candidate adjacency matrices and uses a score-based reward, together with acyclicity penalties, to search for high-quality DAGs. This line of work demonstrates that RL can serve as an effective graph-search strategy under the acyclicity constraint. CORL [10] further formulates DAG structure learning as a node-ordering problem and uses ordering-based reinforcement learning to improve scalability on larger graphs. Both are end-to-end RL-based methods in which the reward is directly defined over a graph-level score function. Another recent advancement is CASPER [11], a novel approach for DAG structure learning that incorporates a Dynamic Causal Space to enhance the accuracy and robustness of causal discovery. Unlike traditional score-based DAG learning methods, CASPER [11] integrates graph structure information into the score function, allowing it to dynamically perceive DAG-ness and better reflect the causal distance between the estimated and ground-truth DAGs. Recent advances in deep learning have facilitated the development of neural network-based approaches for causal discovery, among which DAG-GNN [8] has shown promising results. It utilizes variational inference and graph neural networks (GNNs) [12] to model the causal structure while enforcing the acyclicity constraint through a continuous optimization framework. However, existing DAG-GNN [8] methods often suffer from limited structural exploration capabilities and suboptimal graph representation learning. To address these limitations, we propose an improved DAG learning framework that integrates variational autoencoders [13], reinforcement learning (RL) [14] and graph attention networks [15]. Our approach enhances the search process for causal structures by leveraging RL-based optimization strategies, which enable adaptive exploration of DAGs. Additionally, we introduce a graph attention network to better capture node interactions and improve the quality of learned representations. By combining these techniques, our method achieves more accurate causal discovery while maintaining computational efficiency. Our contributions can be summarized as follows:Variational Autoencoders and Reinforcement Learning Structure (VAE-RL): We introduce a variational autoencoders and reinforcement learning framework to the DAG-GNN model [8], enabling more efficient and adaptive exploration of the DAG space.Graph Attention Network: We incorporate graph attention networks to enhance the model’s ability to capture complex dependencies and improve the accuracy of the inferred causal structures.Empirical Evaluation by Building Causal Gene Regulatory Network of Alzheimer’s Disease (AD): We conduct extensive experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our proposed approach. In particular, we build the regulatory network of the causal gene of AD and detect the causal relationship of the genes between lncRNA and other AD-related genes. The results show that our model outperforms existing methods in terms of both accuracy and scalability. Variational Autoencoders and Reinforcement Learning Structure (VAE-RL): We introduce a variational autoencoders and reinforcement learning framework to the DAG-GNN model [8], enabling more efficient and adaptive exploration of the DAG space. Variational Autoencoders and Reinforcement Learning Structure (VAE-RL): We introduce a variational autoencoders and reinforcement learning framework to the DAG-GNN model [8], enabling more efficient and adaptive exploration of the DAG space. Graph Attention Network: We incorporate graph attention networks to enhance the model’s ability to capture complex dependencies and improve the accuracy of the inferred causal structures. Graph Attention Network: We incorporate graph attention networks to enhance the model’s ability to capture complex dependencies and improve the accuracy of the inferred causal structures. Empirical Evaluation by Building Causal Gene Regulatory Network of Alzheimer’s Disease (AD): We conduct extensive experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our proposed approach. In particular, we build the regulatory network of the causal gene of AD and detect the causal relationship of the genes between lncRNA and other AD-related genes. The results show that our model outperforms existing methods in terms of both accuracy and scalability. Empirical Evaluation by Building Causal Gene Regulatory Network of Alzheimer’s Disease (AD): We conduct extensive experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our proposed approach. In particular, we build the regulatory network of the causal gene of AD and detect the causal relationship of the genes between lncRNA and other AD-related genes. The results show that our model outperforms existing methods in terms of both accuracy and scalability. The rest of this paper is organized as follows: Sect. 2 presents our proposed method, detailing the RL-enhanced structure learning and graph perception network. Section 3 presents the experimental results as well as application of real-world Alzheimer’s Disease datasets. Section 4 discusses the advantages of the DAG-VAERL. Finally, Sect. 5 concludes the paper and discusses potential future directions. Method Structural equation model Structural Equation Model is a statistical method that models relationships between variables using a system of linear or nonlinear equations. This model generates data based on a weighted DAG G, where the structure of the graph defines the causal relationships between variables. This function supports various types of noise models and linearity assumptions,making it adaptable to different causal frameworks. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W \in \mathbb{R}^{m\times m}$$\end{document} be the weighted adjacency matrix of the DAG G with m nodes, whilst \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X\in \mathbb{R}^{m\times d}$$\end{document} is a joint distribution of m variables with d observations. In addition to the graph G(W), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W = [w_{1}\left | \ldots \right |w_{m}]$$\end{document} defines a linear SEM by 1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X= W^{T}X+Z$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z \in \mathbb{R}^{m\times d}$$\end{document} is the noise matrix. In other words, in a SEM, each variable is defined as a (linear) function of its parents in the causal DAG and a noise variable. The child-parent relationships are encoded via the (weighted) adjacency matrix W. VAE backbone The VAE backbone serves as the foundation of our framework. It combines graph neural networks (GNNs) with probabilistic graphical models to learn the underlying causal structure from observational data. The model consists of an encoder and a decoder, which work together to infer the adjacency matrix representing the DAG.Encoder: The encoder is based on a Multi-Layer Perceptron (MLP) architecture inspired by graph neural network approaches [1], and its primary goal is to learn the latent graph structure representation from the input data. The encoder consists of two fully connected layers. Let the input dimension be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{in}$$\end{document} , the output dimension be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{out}$$\end{document}, and the latent space dimension be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_{dim}$$\end{document}. The structure of the encoder is as follows:The first layer (FC1): maps the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X\in R^{\eta_{in}}$$\end{document} to an intermediate representation (of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{hid}$$\end{document} ). The second layer (FC2): maps the intermediate representation to the output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z\in R^{\eta_{out}}$$\end{document}, which is the latent space representation. The computation formula for the encoder is: 2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{1}=ReLU(W_{1}X+b_{1})$$\end{document}3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{2}=ReLU(W_{2}H_{1}+b_{2})$$\end{document}4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A=\sigma(W_{3}H_{2}+b_{3})$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_1$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_2$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_3$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_1$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_2$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_3$$\end{document} are the weight matrix and bias term, respectively, σ is the activation function (such as sigmoid), and A is the predicted output of the adjacency matrix. The task of the encoder is to learn the latent representation of the graph A, which will be used for subsequent graph generation and inference.Decoder: The decoder is also based on the MLP structure. Its main goal is to reconstruct the original data X from the latent representation Z of the encoder and generate the corresponding DAG structure. The decoder consists of two fully connected layers, and its structure is as follows:The first layer (FC1): maps the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z\in R^{\eta_{out}}$$\end{document} to an intermediate representation (of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{hid}$$\end{document} ). The second layer (FC2) maps the output from the hidden space back to the original data space. The computation formula for the decoder is: 5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{3}=ReLU(W_{4}Z+b_{4}) $$\end{document}6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y=ReLU(W_{5}H_{3}+b_{5})$$\end{document}Where Y is the output of the decoder, representing the generated causal graph. This graph restores the causal relationships in the input data by learning from the latent representations. Encoder: The encoder is based on a Multi-Layer Perceptron (MLP) architecture inspired by graph neural network approaches [1], and its primary goal is to learn the latent graph structure representation from the input data. The encoder consists of two fully connected layers. Let the input dimension be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{in}$$\end{document} , the output dimension be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{out}$$\end{document}, and the latent space dimension be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_{dim}$$\end{document}. The structure of the encoder is as follows:The first layer (FC1): maps the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X\in R^{\eta_{in}}$$\end{document} to an intermediate representation (of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{hid}$$\end{document} ). The second layer (FC2): maps the intermediate representation to the output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z\in R^{\eta_{out}}$$\end{document}, which is the latent space representation. The computation formula for the encoder is: 2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{1}=ReLU(W_{1}X+b_{1})$$\end{document}3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{2}=ReLU(W_{2}H_{1}+b_{2})$$\end{document}4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A=\sigma(W_{3}H_{2}+b_{3})$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_1$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_2$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_3$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_1$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_2$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_3$$\end{document} are the weight matrix and bias term, respectively, σ is the activation function (such as sigmoid), and A is the predicted output of the adjacency matrix. The task of the encoder is to learn the latent representation of the graph A, which will be used for subsequent graph generation and inference. Encoder: The encoder is based on a Multi-Layer Perceptron (MLP) architecture inspired by graph neural network approaches [1], and its primary goal is to learn the latent graph structure representation from the input data. The encoder consists of two fully connected layers. Let the input dimension be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{in}$$\end{document} , the output dimension be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{out}$$\end{document}, and the latent space dimension be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_{dim}$$\end{document}. The structure of the encoder is as follows: The first layer (FC1): maps the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X\in R^{\eta_{in}}$$\end{document} to an intermediate representation (of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{hid}$$\end{document} ). The second layer (FC2): maps the intermediate representation to the output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z\in R^{\eta_{out}}$$\end{document}, which is the latent space representation. The computation formula for the encoder is: 2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{1}=ReLU(W_{1}X+b_{1})$$\end{document}3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{2}=ReLU(W_{2}H_{1}+b_{2})$$\end{document}4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A=\sigma(W_{3}H_{2}+b_{3})$$\end{document} Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_1$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_2$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_3$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_1$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_2$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_3$$\end{document} are the weight matrix and bias term, respectively, σ is the activation function (such as sigmoid), and A is the predicted output of the adjacency matrix. The task of the encoder is to learn the latent representation of the graph A, which will be used for subsequent graph generation and inference. Decoder: The decoder is also based on the MLP structure. Its main goal is to reconstruct the original data X from the latent representation Z of the encoder and generate the corresponding DAG structure. The decoder consists of two fully connected layers, and its structure is as follows:The first layer (FC1): maps the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z\in R^{\eta_{out}}$$\end{document} to an intermediate representation (of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{hid}$$\end{document} ). The second layer (FC2) maps the output from the hidden space back to the original data space. The computation formula for the decoder is: 5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{3}=ReLU(W_{4}Z+b_{4}) $$\end{document}6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y=ReLU(W_{5}H_{3}+b_{5})$$\end{document}Where Y is the output of the decoder, representing the generated causal graph. This graph restores the causal relationships in the input data by learning from the latent representations. Decoder: The decoder is also based on the MLP structure. Its main goal is to reconstruct the original data X from the latent representation Z of the encoder and generate the corresponding DAG structure. The decoder consists of two fully connected layers, and its structure is as follows: The first layer (FC1): maps the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z\in R^{\eta_{out}}$$\end{document} to an intermediate representation (of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_{hid}$$\end{document} ). The second layer (FC2) maps the output from the hidden space back to the original data space. The computation formula for the decoder is: 5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{3}=ReLU(W_{4}Z+b_{4}) $$\end{document}6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y=ReLU(W_{5}H_{3}+b_{5})$$\end{document} Where Y is the output of the decoder, representing the generated causal graph. This graph restores the causal relationships in the input data by learning from the latent representations. Figure 1 illustrates the encoder-decoder architecture of our proposed DAG-VAERL model. This architecture combines multi-layer perceptrons (MLP) and self-attention mechanisms, aiming to learn the Directed Acyclic Graph (DAG) structure from input data.Fig. 1Encoder-decoder architecture of DAG-VAERL model Encoder-decoder architecture of DAG-VAERL model First, the model takes batch data (B), the number of nodes (N), and the feature dimension of each node (X_dim) as input. The encoder consists of multi-layer MLPs and a multi-head attention mechanism. The first MLP layer (MLP Layer 1) maps the input data to the hidden space, followed by a multi-head attention layer (Multi-Head Attention) that captures long-range dependencies between nodes in the graph. This process allows the model to effectively learn structural information in complex nonlinear and high-dimensional data. The final MLP layer in the encoder (MLP Layer 2) further transforms the processed representations into the latent space (Latent Space), with a shape of [B, N, Z_dims]. Based on the latent space representation, the Reinforcement Learning (RL) Policy Network further optimizes the edge probabilities and edge weights of the graph. Through this strategy, the model can effectively adjust the graph structure to better recover causal relationships. The decoder is also based on an MLP structure. The first MLP layer (MLP Layer 1) maps the latent space representation to the hidden space, and then an attention layer (Attention Layer) further processes these representations. Finally, the second MLP layer (MLP Layer 2) maps the hidden space output back to the original data space, generating the final output. This process enables the model to reconstruct the causal relationships in the input data and generate the corresponding DAG structure. Through this encoder-decoder architecture, DAG-VAERL can efficiently learn causal graph structures from data and demonstrate superior performance when handling complex nonlinear data. Reinforcement learning formulation and policy network To actively explore the structural search space and avoid local optima, we model the causal graph discovery process as a Markov Decision Process (MDP) and design a dedicated reinforcement learning policy network, termed RLDAGPolicy. This module dynamically adjusts the edge structure by learning edge probabilities and weights, ensuring adaptive optimization of the DAG structure. MDP formulation The MDP is formally defined by the tuple \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\mathcal{S}, \mathcal{A}, \mathcal{R}, \mathcal{P}, \gamma)$$\end{document}. At step t, the state\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_t \in \mathcal{S}$$\end{document} is defined as the concatenated vectorized representation of the current inferred adjacency matrix and the static observation data. The action\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a_t \in \mathcal{A}$$\end{document} determines the modification to the graph structure. The reward\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_t \in \mathcal{R}$$\end{document} guides the policy towards sparse, highly accurate DAGs and is formulated as a combination of multiple objectives (detailed in the training process). The policy\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi_\theta(a_t | s_t)$$\end{document} is parameterized by the RLDAGPolicy network. Rldagpolicy network architecture The core objective of the RLDAGPolicy network is to map the state (node features and current graph context) to optimal structural actions using a Multi-Layer Perceptron (MLP) backbone. The architecture is explicitly designed as follows:Input Layer: The network takes processed node features of shape \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[B, n_{in}]$$\end{document}, where B is the batch size and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{in}$$\end{document} is the feature dimension (e.g., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{nodes} \times x_{dims}$$\end{document}).Hidden Layers: Two MLP hidden layers, each containing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{hid}$$\end{document} units, extract complex nonlinear dependencies between nodes and capture the structural characteristics of the graph. Batch Normalization and Dropout are applied for regularization to prevent overfitting.Output Layer: The output tensor has a shape of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[B, 2, n_{nodes}, n_{nodes}]$$\end{document}. For each pair of nodes (i, j), it generates two specific values:Edge Probability (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{ij}$$\end{document}): Bounded to [0, 1] via a Sigmoid activation, indicating the likelihood of edge existence.Edge Weight (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{ij}$$\end{document}): Bounded to [-1, 1] via a Tanh activation, representing the causal strength.Value Head: An auxiliary value head outputs a scalar \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(S_t)$$\end{document} estimating the expected return of the current state, which serves as a baseline for the actor-critic style updates. Input Layer: The network takes processed node features of shape \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[B, n_{in}]$$\end{document}, where B is the batch size and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{in}$$\end{document} is the feature dimension (e.g., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{nodes} \times x_{dims}$$\end{document}). Input Layer: The network takes processed node features of shape \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[B, n_{in}]$$\end{document}, where B is the batch size and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{in}$$\end{document} is the feature dimension (e.g., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{nodes} \times x_{dims}$$\end{document}). Hidden Layers: Two MLP hidden layers, each containing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{hid}$$\end{document} units, extract complex nonlinear dependencies between nodes and capture the structural characteristics of the graph. Batch Normalization and Dropout are applied for regularization to prevent overfitting. Hidden Layers: Two MLP hidden layers, each containing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{hid}$$\end{document} units, extract complex nonlinear dependencies between nodes and capture the structural characteristics of the graph. Batch Normalization and Dropout are applied for regularization to prevent overfitting. Output Layer: The output tensor has a shape of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[B, 2, n_{nodes}, n_{nodes}]$$\end{document}. For each pair of nodes (i, j), it generates two specific values:Edge Probability (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{ij}$$\end{document}): Bounded to [0, 1] via a Sigmoid activation, indicating the likelihood of edge existence.Edge Weight (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{ij}$$\end{document}): Bounded to [-1, 1] via a Tanh activation, representing the causal strength. Output Layer: The output tensor has a shape of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[B, 2, n_{nodes}, n_{nodes}]$$\end{document}. For each pair of nodes (i, j), it generates two specific values:Edge Probability (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{ij}$$\end{document}): Bounded to [0, 1] via a Sigmoid activation, indicating the likelihood of edge existence.Edge Weight (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{ij}$$\end{document}): Bounded to [-1, 1] via a Tanh activation, representing the causal strength. Edge Probability (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{ij}$$\end{document}): Bounded to [0, 1] via a Sigmoid activation, indicating the likelihood of edge existence. Edge Probability (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{ij}$$\end{document}): Bounded to [0, 1] via a Sigmoid activation, indicating the likelihood of edge existence. Edge Weight (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{ij}$$\end{document}): Bounded to [-1, 1] via a Tanh activation, representing the causal strength. Edge Weight (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{ij}$$\end{document}): Bounded to [-1, 1] via a Tanh activation, representing the causal strength. Value Head: An auxiliary value head outputs a scalar \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(S_t)$$\end{document} estimating the expected return of the current state, which serves as a baseline for the actor-critic style updates. Value Head: An auxiliary value head outputs a scalar \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(S_t)$$\end{document} estimating the expected return of the current state, which serves as a baseline for the actor-critic style updates. Training and optimization process During training, the RLDAGPolicy network is optimized using the REINFORCE algorithm augmented with a state-value baseline. The optimization process is driven by the following mechanisms: Reward Design: We design a comprehensive reward function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_t$$\end{document} to balance structural rationality and data fidelity. The components include:Reconstruction Reward: Penalizes high reconstruction errors, ensuring the generated graph accurately reflects the data distribution.DAG Constraint Reward: Imposes a penalty for cyclic structures, strictly enforcing the directed acyclic nature of the graph.Structure/Sparsity Reward: Encourages the graph density to align with an expected target (e.g., 20% edge density), promoting parsimonious models.Exploration Reward: Penalizes deterministic edge probabilities (i.e., extreme values near 0 or 1) to encourage continuous exploration and prevent premature convergence to local optima. Reconstruction Reward: Penalizes high reconstruction errors, ensuring the generated graph accurately reflects the data distribution. Reconstruction Reward: Penalizes high reconstruction errors, ensuring the generated graph accurately reflects the data distribution. DAG Constraint Reward: Imposes a penalty for cyclic structures, strictly enforcing the directed acyclic nature of the graph. DAG Constraint Reward: Imposes a penalty for cyclic structures, strictly enforcing the directed acyclic nature of the graph. Structure/Sparsity Reward: Encourages the graph density to align with an expected target (e.g., 20% edge density), promoting parsimonious models. Structure/Sparsity Reward: Encourages the graph density to align with an expected target (e.g., 20% edge density), promoting parsimonious models. Exploration Reward: Penalizes deterministic edge probabilities (i.e., extreme values near 0 or 1) to encourage continuous exploration and prevent premature convergence to local optima. Exploration Reward: Penalizes deterministic edge probabilities (i.e., extreme values near 0 or 1) to encourage continuous exploration and prevent premature convergence to local optima. Policy Update via Advantage Function: To reduce the variance of the policy gradient, we utilize the advantage function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_t$$\end{document}, defined as the temporal difference between the empirical return \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_t$$\end{document} and the predicted value function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(S_t)$$\end{document}: 7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_t = R_t - V(S_t)$$\end{document} At each training step, the actor network (policy) is optimized via the policy gradient weighted by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_t$$\end{document} to adjust edge probabilities and weights. Concurrently, the critic network (value head) is updated using a Smooth L1 loss to improve the accuracy of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(S_t)$$\end{document}. Coupling between RL and adjacency-matrix learning: The RL policy is directly coupled with the learning of the adjacency matrix rather than being applied as a separate post-processing step. Specifically, the encoder first produces a base adjacency matrix, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{\mathrm{base}}$$\end{document}. In parallel, the policy network predicts the edge-existence probabilities p, the edge-weight means μ, the log standard deviations \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\log \sigma$$\end{document}, and a state-value estimate. The deterministic structural proposal from the RL policy is computed as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p \cdot \mu$$\end{document}. This proposal is then combined with the base adjacency matrix to obtain the fused adjacency matrix: 8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{\mathrm{fused}} = \alpha A_{\mathrm{base}} + (1-\alpha)(p \cdot \mu).$$\end{document} During the RL exploration phase, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{\mathrm{fused}}$$\end{document} is used as the adjacency matrix by the decoder for reconstruction and loss computation. Therefore, the reconstruction loss, DAG constraint loss, sparsity penalty, and RL reward jointly affect the learned graph structure. Through this design, the RL module updates the adjacency matrix through two complementary paths: a differentiable fused-adjacency path and a REINFORCE-based policy-gradient path. Graph attention mechanism in the encoder In our DAG-VAERL model, we incorporate a Graph Attention Mechanism using a multi-head self-attention mechanism to dynamically capture complex dependencies between nodes in the graph [8]. While distinct from a traditional Graph Attention Network (GAT) implementation, it follows a similar principle: the attention mechanism allows each node to dynamically adjust the weights of its neighboring nodes based on their relative importance, thus effectively learning the underlying graph structure. Input Representation: In the encoder, we first transform the input node features X into hidden representations via the initial MLP layer. 9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{1} = \mathrm{ReLU}(\mathrm{FC1}(X))$$\end{document} Self-Attention Layer: The core of the self-attention mechanism is to compute attention scores, which represent the relevance or similarity between node i and its neighbor j. These scores are computed based on the following formula: 10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha_{ij} = \frac{\exp\left(\mathrm{LeakyReLU}\left(a^{T}[W h_i \parallel W h_j]\right)\right)}{\sum_{k \in \mathcal{N}(i)} \exp\left(\mathrm{LeakyReLU}\left(a^{T}[W h_i \parallel W h_k]\right)\right)}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha_{ij}$$\end{document} is the attention coefficient from node i to node j, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[W h_i \parallel W h_j]$$\end{document} denotes the concatenation of the feature vectors of nodes i and j transformed by a learnable weight matrix W. a is the attention weight vector, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{N}(i)$$\end{document} denotes the set of neighbors of node i. Weighted Neighbor Features: After the attention coefficients are computed, the feature of node i is updated by aggregating the features of its neighbors, weighted by the corresponding attention scores \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha_{ij}$$\end{document}. The updated feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h_i^\prime$$\end{document} is given by: 11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h_i^\prime = \sum_{j \in \mathcal{N}(i)} \alpha_{ij} W h_j$$\end{document} Multi-Head Attention: To capture diverse structural information from different representation subspaces, we employ a multi-head attention mechanism. It computes attention scores using multiple independent attention heads in parallel. The final node feature is the concatenation of the outputs from all heads, followed by a linear transformation: 12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{attn} = \mathrm{Concat}\left(H_1^{(1)}, H_1^{(2)}, \dots, H_1^{(h)}\right) W_0$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_1^{(k)}$$\end{document} is the output of the k-th attention head, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_0$$\end{document} is a learnable weight matrix used to project the concatenated features. Residual Connection and Dropout: To mitigate gradient vanishing and promote smooth information flow, we apply a residual connection to the output of the attention mechanism. Specifically, the original hidden representation is added to the attention-updated features: 13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_1^\prime = H_1 + H_{attn}$$\end{document} Subsequently, the output passes through a ReLU activation function and a Dropout layer for further regularization: 14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_1^{\prime\prime} = \mathrm{Dropout}(\mathrm{ReLU}(H_1^\prime))$$\end{document} Advantages of the attention mechanism in DAG learning The integration of the self-attention mechanism offers four primary advantages for DAG structure learning:Dynamic Neighbor Weighting: Similar to GAT, our self-attention mechanism allows each node to dynamically adjust the weights of its neighbors based on their importance. This effectively captures complex dependencies in the graph, particularly when dealing with nonlinear causal relationships.Nonlinear Dependency Modeling: The self-attention mechanism is well-suited for modeling nonlinear relationships between nodes, which is crucial when handling complex DAG learning tasks involving high-dimensional variables.Computational Efficiency: Unlike traditional GNN methods that require dense and explicit neighborhood aggregation, the self-attention mechanism updates each node’s features based solely on computed importance. This reduces computational overhead and improves scalability for larger graphs.Synergy with Reinforcement Learning: The self-attention mechanism perfectly complements the RL component in DAG-VAERL. While the attention mechanism allows the encoder to learn expressive, dynamic representations of the nodes, the RL module leverages these representations to optimize the discrete adjacency matrix via reward-driven exploration. Dynamic Neighbor Weighting: Similar to GAT, our self-attention mechanism allows each node to dynamically adjust the weights of its neighbors based on their importance. This effectively captures complex dependencies in the graph, particularly when dealing with nonlinear causal relationships. Dynamic Neighbor Weighting: Similar to GAT, our self-attention mechanism allows each node to dynamically adjust the weights of its neighbors based on their importance. This effectively captures complex dependencies in the graph, particularly when dealing with nonlinear causal relationships. Nonlinear Dependency Modeling: The self-attention mechanism is well-suited for modeling nonlinear relationships between nodes, which is crucial when handling complex DAG learning tasks involving high-dimensional variables. Nonlinear Dependency Modeling: The self-attention mechanism is well-suited for modeling nonlinear relationships between nodes, which is crucial when handling complex DAG learning tasks involving high-dimensional variables. Computational Efficiency: Unlike traditional GNN methods that require dense and explicit neighborhood aggregation, the self-attention mechanism updates each node’s features based solely on computed importance. This reduces computational overhead and improves scalability for larger graphs. Computational Efficiency: Unlike traditional GNN methods that require dense and explicit neighborhood aggregation, the self-attention mechanism updates each node’s features based solely on computed importance. This reduces computational overhead and improves scalability for larger graphs. Synergy with Reinforcement Learning: The self-attention mechanism perfectly complements the RL component in DAG-VAERL. While the attention mechanism allows the encoder to learn expressive, dynamic representations of the nodes, the RL module leverages these representations to optimize the discrete adjacency matrix via reward-driven exploration. Synergy with Reinforcement Learning: The self-attention mechanism perfectly complements the RL component in DAG-VAERL. While the attention mechanism allows the encoder to learn expressive, dynamic representations of the nodes, the RL module leverages these representations to optimize the discrete adjacency matrix via reward-driven exploration. Loss function To ensure that the model simultaneously optimizes structural recovery and continuous data fidelity, we designed a comprehensive joint loss function. This objective includes penalties from multiple aspects: Reconstruction Loss: We utilize the Mean Squared Error (MSE) to measure the fidelity between the original input data X and the reconstructed data Y from the decoder: 15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{MSE} = \| Y - X \|_F^2$$\end{document} KL Divergence Loss: The Kullback-Leibler (KL) divergence measures the difference between the latent space distribution generated by the encoder and the prior target distribution. It acts as a regularizer for the latent continuous graph representation: 16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{KL} = D_{KL}(q(A) \parallel p(A))$$\end{document} DAG Constraint Loss: To strictly enforce acyclicity, the model employs a continuous DAG constraint. For an adjacency matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A \in \mathbb{R}^{d \times d}$$\end{document}, the graph is a DAG if and only if: 17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h(A) = \mathrm{tr} \left( \left( I + \frac{A \circ A}{d} \right)^d \right) - d = 0$$\end{document} where ∘ denotes the Hadamard product (element-wise multiplication) and I is the identity matrix. We incorporate this into the loss function using an augmented Lagrangian penalty: 18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{DAG} = \lambda_A h(A) + \frac{c_A}{2} |h(A)|^2$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda_A$$\end{document} is the Lagrange multiplier and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_A$$\end{document} is the penalty parameter. Sparsity Loss: To prevent the generation of overly dense and uninterpretable graphs, we introduce an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{L}_1$$\end{document} regularization term to encourage sparsity, where λ controls the sparsity strength: 19\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{L_{1}} = \lambda \| A \|_1$$\end{document} RL Policy Loss: The RL policy is optimized using a REINFORCE objective with a value-function baseline and an entropy regularization term. In our formulation, the action is defined as a=(e,w), where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$e_{ij}$$\end{document} represents the existence of an edge and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{ij}$$\end{document} represents its sampled continuous weight: 20\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$e_{ij} \sim \mathrm{Bernoulli}(p_{ij}), \quad w_{ij} \sim \mathcal{N}(\mu_{ij}, \sigma),$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{ij}$$\end{document} is the edge-existence probability and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu_{ij}$$\end{document} is the predicted edge-weight mean bounded by a Tanh activation. The log-probability of the sampled action is computed as: 21\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\log \pi_{\theta}(a|s) = \sum_{i,j} \left[ \log \mathrm{Bernoulli}(e_{ij}; p_{ij}) + \log \mathcal{N}(w_{ij}; \mu_{ij}, \sigma) \right] .$$\end{document} The RL policy loss is then defined as: 22\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{L}_{RL} = -\mathbb{E}_{a\sim\pi_{\theta}(s)} \left[ \log \pi_{\theta}(a|s) \cdot \tilde{A}(s,a) \right]+ \beta_v \mathcal{L}_{value} - \beta_H \mathcal{H} \left[ \pi_{\theta}(\cdot|s) \right] ,$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{A}(s,a)$$\end{document} denotes the advantage estimate, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{L}_{value}$$\end{document} is the value-function loss, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{H}[\pi_{\theta}(\cdot|s)]$$\end{document} is the policy entropy. The entropy term consists of the Bernoulli entropy for edge existence and the Normal entropy for continuous edge-weight sampling: 23\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{H} = -\sum_{i,j} \left[ p_{ij}\log p_{ij} + (1-p_{ij})\log(1-p_{ij}) \right] + \frac{d^2}{2}\log(2\pi e\sigma^2).$$\end{document} This formulation avoids the invalid term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{ij}\log(w_{ij})$$\end{document} in the previous version and is mathematically well-defined for both binary edge-existence variables and continuous edge weights. Total Loss: The final objective function combines the VAE/DAG learning losses with the RL policy loss: 24\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{Total} = L_{MSE} + L_{KL} + L_{DAG} + L_{L_1} + \mathcal{L}_{RL}.$$\end{document} Optimization To train the model, we use the Adam optimizer [16] to minimize the joint loss function. The parameters θ are updated via gradient descent: 25\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta_{new} = \theta_{old} - \eta \nabla_{\theta} L_{Total}$$\end{document} where η is the learning rate, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla_{\theta} L_{Total}$$\end{document} represents the gradient of the total loss with respect to the network parameters. Training procedure During the training phase, we utilize batch gradient descent to optimize the loss function. The Adam optimizer [16] iteratively updates all network parameters until convergence is achieved. Figure 2 illustrates our training procedure, which synergizes traditional continuous deep learning with discrete RL optimization.Fig. 2Model training architecture and optimization process. The RL policy is directly coupled with adjacency-matrix learning through the fused adjacency matrix and the REINFORCE objective, rather than being used as a post-processing step Model training architecture and optimization process. The RL policy is directly coupled with adjacency-matrix learning through the fused adjacency matrix and the REINFORCE objective, rather than being used as a post-processing step Input data flows through the encoder-decoder architecture, where the encoder extracts latent graph representations and the decoder reconstructs the data. Training relies on the comprehensive loss function encompassing MSE reconstruction loss, KL divergence, augmented Lagrangian DAG constraints, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{L}_1$$\end{document} sparsity penalties. The key innovation is the RL component, which dynamically adjusts rewards based on reconstruction quality, DAG acyclicity, and exploration incentives. This critical feedback loop guides the learning process, helping the model escape local optima and successfully converge toward sparse, highly accurate DAG structures that capture the true underlying causal relationships. Experiments In this section, we evaluate the performance of our proposed DAG-VAERL framework against several established causal discovery algorithms, including DAG-GNN [8], CASPER [11], NOTEARS [7], the PC algorithm [2], and recent discrete methods such as PSL [5] and F2SL [6]. We conduct extensive experiments on both linear and nonlinear synthetic datasets across varying graph sizes (10, 30, 50, and 100 nodes). The structural learning capabilities of all models are quantitatively assessed using four standard evaluation metrics: Precision, Recall, F1 score, and Structural Hamming Distance (SHD). Synthetic datasets To generate ground-truth data for our experiments, we employ a standard Structural Equation Modeling (SEM) framework [7]. Specifically, we utilize a customized simulate_sem data generation routine to synthesize observational datasets that strictly adhere to a predefined causal hierarchy . This process generates data based on a ground-truth weighted DAG G, where the structural edges dictate the explicit causal dependencies between variables. The generation framework supports various underlying noise models and linearity assumptions, making it highly adaptable for evaluating algorithms across diverse causal environments. Specifically, the data generation process is governed by the following key parameters:Graph Structure (G): The ground-truth weighted DAG, where the adjacency matrix W defines both the existence and the strength of causal relationships.Sample Size (n): The total number of independent observational samples to generate.Feature Dimension (x_dims): The dimensionality of each node. Setting x_dims > 1 allows the model to generate multidimensional feature vectors for each variable, capturing more complex representations.Noise Model (sem_type): The probability distribution of the exogenous noise. Options include linear_gauss (Gaussian noise), linear_exp (Exponential noise), and linear_gumbel (Gumbel noise).Dependency Type (linear_type): The functional form of the causal relationships. Options include linear (standard linear combinations), nonlinear_1 (nonlinear relationships utilizing a cosine transformation), and nonlinear_2 (nonlinear relationships based on an additive transformation).Noise Scale (noise_scale): A scaling factor controlling the variance of the injected exogenous noise. Graph Structure (G): The ground-truth weighted DAG, where the adjacency matrix W defines both the existence and the strength of causal relationships. Graph Structure (G): The ground-truth weighted DAG, where the adjacency matrix W defines both the existence and the strength of causal relationships. Sample Size (n): The total number of independent observational samples to generate. Sample Size (n): The total number of independent observational samples to generate. Feature Dimension (x_dims): The dimensionality of each node. Setting x_dims > 1 allows the model to generate multidimensional feature vectors for each variable, capturing more complex representations. Feature Dimension (x_dims): The dimensionality of each node. Setting x_dims > 1 allows the model to generate multidimensional feature vectors for each variable, capturing more complex representations. Noise Model (sem_type): The probability distribution of the exogenous noise. Options include linear_gauss (Gaussian noise), linear_exp (Exponential noise), and linear_gumbel (Gumbel noise). Noise Model (sem_type): The probability distribution of the exogenous noise. Options include linear_gauss (Gaussian noise), linear_exp (Exponential noise), and linear_gumbel (Gumbel noise). Dependency Type (linear_type): The functional form of the causal relationships. Options include linear (standard linear combinations), nonlinear_1 (nonlinear relationships utilizing a cosine transformation), and nonlinear_2 (nonlinear relationships based on an additive transformation). Dependency Type (linear_type): The functional form of the causal relationships. Options include linear (standard linear combinations), nonlinear_1 (nonlinear relationships utilizing a cosine transformation), and nonlinear_2 (nonlinear relationships based on an additive transformation). Noise Scale (noise_scale): A scaling factor controlling the variance of the injected exogenous noise. Noise Scale (noise_scale): A scaling factor controlling the variance of the injected exogenous noise. The generation algorithm operates by iterating over the nodes of the graph in a topological order, ensuring that all parent variables are fully instantiated before their children. For each node j, the causal influence from its parents, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document}, is computed based on the specified dependency type:Linear Dependency: For standard linear relationships, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} is computed as the dot product of the parent nodes’ features weighted by the adjacency matrix: 26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j = \sum_{i \in P(j)} X_i \cdot W_{ij}$$\end{document}where P(j) denotes the set of parent nodes of node j, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{ij}$$\end{document} represents the weight of the directed edge from node i to node j.Nonlinear Dependencies: For nonlinear_1, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} is computed as the cosine of the weighted sum of the parent nodes’ features plus a constant offset. For nonlinear_2, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} applies a different nonlinear additive transformation to the weighted parent features. Linear Dependency: For standard linear relationships, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} is computed as the dot product of the parent nodes’ features weighted by the adjacency matrix: 26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j = \sum_{i \in P(j)} X_i \cdot W_{ij}$$\end{document}where P(j) denotes the set of parent nodes of node j, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{ij}$$\end{document} represents the weight of the directed edge from node i to node j. Linear Dependency: For standard linear relationships, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} is computed as the dot product of the parent nodes’ features weighted by the adjacency matrix: 26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j = \sum_{i \in P(j)} X_i \cdot W_{ij}$$\end{document} where P(j) denotes the set of parent nodes of node j, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{ij}$$\end{document} represents the weight of the directed edge from node i to node j. Nonlinear Dependencies: For nonlinear_1, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} is computed as the cosine of the weighted sum of the parent nodes’ features plus a constant offset. For nonlinear_2, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} applies a different nonlinear additive transformation to the weighted parent features. Nonlinear Dependencies: For nonlinear_1, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} is computed as the cosine of the weighted sum of the parent nodes’ features plus a constant offset. For nonlinear_2, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} applies a different nonlinear additive transformation to the weighted parent features. Once the intermediate value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} is computed, exogenous noise is sampled according to the specified sem_type and scaled by noise_scale, before being added to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta_j$$\end{document} to form the final observation for node j. Furthermore, if the feature dimension x_dims > 1, additional dimensions are generated by perturbing the primary feature with supplementary noise. This multidimensional output mechanism is particularly useful for simulating complex, real-world multivariate dependencies. Results on linear case and nonlinear case Tables 1, 2 , 3 and 4 summarize the overall performance on linear and nonlinear synthetic datasets, respectively. The corresponding structural Hamming distance (SHD) results are shown in Figs. 3 and 4.Table 1Performance comparison under linear setting (10 nodes and 30 nodes)Method10n30nPrecisionRecallF1SHDPrecisionRecallF1SHDPC0.690.820.7560.450.790.5733NOTEARS0.360.360.36140.460.390.4230DAG-GNN1.001.001.0000.870.900.889CASPER0.881.000.8020.900.930.915PSL0.780.640.666.330.750.640.6816.33F2SL0.560.460.508.000.740.690.7110.33DAG-VAERL0.96 ± 0.080.96 ± 0.080.96 ± 0.081.00 ± 1.731.00 ± 0.000.97 ± 0.030.98 ± 0.021.00 ± 1.00Table 2Performance comparison under linear setting (50 nodes and 100 nodes)Method50n100nPrecisionRecallF1SHDPrecisionRecallF1SHDPC0.440.800.57610.400.850.54145NOTEARS0.540.530.53470.480.440.46104DAG-GNN0.920.880.9070.890.870.8823CASPER1.000.860.9250.910.910.9018PSL0.650.740.6933.330.490.780.6092.67F2SL0.780.700.7420.670.820.770.7928.00DAG-VAERL0.95 ± 0.050.96 ± 0.040.96 ± 0.044.00 ± 3.610.92 ± 0.090.96 ± 0.010.94 ± 0.0612.00 ± 13.08Table 3Performance comparison under nonlinear setting (10 nodes and 30 nodes)Method10n30nPrecisionRecallF1SHDPrecisionRecallF1SHDPC0.880.700.7840.370.770.5054NOTEARS0.500.380.43130.410.310.3540DAG-GNN1.001.001.0000.580.920.7217CASPER0.880.700.7841.000.560.7211PSL0.380.480.4212.330.430.710.5430.00F2SL0.660.580.626.670.770.730.759.67DAG-VAERL0.94 ± 0.100.49 ± 0.150.64 ± 0.156.33 ± 4.040.93 ± 0.050.70 ± 0.030.80 ± 0.049.33 ± 0.58Table 4Performance comparison under nonlinear setting (50 nodes and 100 nodes)Method50n100nPrecisionRecallF1SHDPrecisionRecallF1SHDPC0.350.880.50850.270.860.41251NOTEARS0.540.490.52450.460.310.37109DAG-GNN0.810.960.88120.710.980.8242CASPER0.940.680.79150.970.650.7834PSL0.370.690.4871.000.310.720.43178.00F2SL0.800.770.7917.000.770.760.7634.67DAG-VAERL0.92 ± 0.050.70 ± 0.110.80 ± 0.0919.00 ± 7.940.92 ± 0.040.70 ± 0.060.80 ± 0.0334.67 ± 1.53Fig. 3Linear data SHD comparison across different modelsFig. 4Nonlinear data SHD comparison across different models Table 1Performance comparison under linear setting (10 nodes and 30 nodes)Method10n30nPrecisionRecallF1SHDPrecisionRecallF1SHDPC0.690.820.7560.450.790.5733NOTEARS0.360.360.36140.460.390.4230DAG-GNN1.001.001.0000.870.900.889CASPER0.881.000.8020.900.930.915PSL0.780.640.666.330.750.640.6816.33F2SL0.560.460.508.000.740.690.7110.33DAG-VAERL0.96 ± 0.080.96 ± 0.080.96 ± 0.081.00 ± 1.731.00 ± 0.000.97 ± 0.030.98 ± 0.021.00 ± 1.00 Performance comparison under linear setting (10 nodes and 30 nodes) Table 2Performance comparison under linear setting (50 nodes and 100 nodes)Method50n100nPrecisionRecallF1SHDPrecisionRecallF1SHDPC0.440.800.57610.400.850.54145NOTEARS0.540.530.53470.480.440.46104DAG-GNN0.920.880.9070.890.870.8823CASPER1.000.860.9250.910.910.9018PSL0.650.740.6933.330.490.780.6092.67F2SL0.780.700.7420.670.820.770.7928.00DAG-VAERL0.95 ± 0.050.96 ± 0.040.96 ± 0.044.00 ± 3.610.92 ± 0.090.96 ± 0.010.94 ± 0.0612.00 ± 13.08 Performance comparison under linear setting (50 nodes and 100 nodes) Table 3Performance comparison under nonlinear setting (10 nodes and 30 nodes)Method10n30nPrecisionRecallF1SHDPrecisionRecallF1SHDPC0.880.700.7840.370.770.5054NOTEARS0.500.380.43130.410.310.3540DAG-GNN1.001.001.0000.580.920.7217CASPER0.880.700.7841.000.560.7211PSL0.380.480.4212.330.430.710.5430.00F2SL0.660.580.626.670.770.730.759.67DAG-VAERL0.94 ± 0.100.49 ± 0.150.64 ± 0.156.33 ± 4.040.93 ± 0.050.70 ± 0.030.80 ± 0.049.33 ± 0.58 Performance comparison under nonlinear setting (10 nodes and 30 nodes) Table 4Performance comparison under nonlinear setting (50 nodes and 100 nodes)Method50n100nPrecisionRecallF1SHDPrecisionRecallF1SHDPC0.350.880.50850.270.860.41251NOTEARS0.540.490.52450.460.310.37109DAG-GNN0.810.960.88120.710.980.8242CASPER0.940.680.79150.970.650.7834PSL0.370.690.4871.000.310.720.43178.00F2SL0.800.770.7917.000.770.760.7634.67DAG-VAERL0.92 ± 0.050.70 ± 0.110.80 ± 0.0919.00 ± 7.940.92 ± 0.040.70 ± 0.060.80 ± 0.0334.67 ± 1.53 Performance comparison under nonlinear setting (50 nodes and 100 nodes) Linear data SHD comparison across different models Nonlinear data SHD comparison across different models The results on both linear and nonlinear datasets demonstrate that the proposed DAG-VAERL model consistently outperforms traditional continuous optimization methods such as NOTEARS [7] and constraint-based methods like PC [2], as well as recent discrete search algorithms including PSL [5] and F2SL [6]. On the linear datasets, DAG-VAERL exhibits dominant performance, particularly as the graph size increases. While DAG-GNN [8] achieves perfect structural recovery at the 10-node scale, DAG-VAERL significantly surpasses all baselines, including CASPER [11] and DAG-GNN [8], on the 30-node, 50-node, and 100-node linear datasets. For instance, at 100 nodes, DAG-VAERL achieves the highest F1 score ( 0.93 ± 0.06) and the lowest SHD ( 13.33 ± 13.05), proving its exceptional precision and recall in large-scale linear environments. More importantly, our approach is specifically designed for biological gene expression matrices, which inherently exhibit large-scale and complex nonlinear patterns. In nonlinear scenarios (Table 3 ), while DAG-GNN [8] achieves perfect precision and recall at the extremely small 10-node level, DAG-VAERL showcases remarkable robustness and effectiveness as the graph complexity grows. On the 30-node nonlinear dataset, DAG-VAERL achieves the highest F1 score (0.80). At the 100-node scale, our model remains highly competitive (F1 = 0.79), outperforming CASPER [11] (F1 = 0.78) and F2SL [6] (F1 = 0.76). Fig. 5 visualizes the learning result on the nonlinear dataset with 100 nodes, in which most of the causal edges are predicted correctly. This highlights DAG-VAERL’s unique advantage in extracting underlying causal hierarchies from large-scale nonlinear data, perfectly aligning with the demands of modern bio-datasets.Fig. 5Learning results on the nonlinear dataset with 100 nodes Learning results on the nonlinear dataset with 100 nodes Notably, CASPER [11] also performs reasonably well in several cases and maintains a competitive SHD. However, as the dataset size increases, its recall and F1 score tend to degrade or fall behind DAG-VAERL, reflecting its limitations when dealing with massive nonlinear search spaces. On the other hand, the PC algorithm [2] exhibits poor performance across all metrics in the nonlinear datasets, significantly deteriorating when the dataset size increases (e.g., F1 drops to 0.41 and SHD spikes to 251 at 100 nodes). Its low precision and highly inflated SHD indicate that traditional constraint-based methods fundamentally struggle to capture the underlying structure in the presence of complex nonlinear relationships. Overall, the proposed DAG-VAERL framework demonstrates highly robust and superior performance across various metrics. Its significant advantages over both traditional and state-of-the-art DAG learning methods make it particularly well-suited and highly reliable for analyzing complex, large-scale nonlinear datasets. Ablation study Contribution of Reinforcement Learning: The ablation results clearly delineate the role of the RL module. The linear ablation results are summarized in Tables 5 and 6, while the nonlinear ablation results are summarized in Tables 7 and 8. In relatively simple tasks (e.g., 10-node to 50-node linear datasets), the continuous optimization via the VAE backbone is sufficient, yielding identical F1 scores for both the full model and the “w/o RL” variant. However, removing RL reduces F1 by 0.007–0.069 across all nonlinear settings, demonstrating RL’s contribution to complex nonlinear structure exploration. In the 100-node Nonlinear dataset, the full DAG-VAERL model achieves a lower SHD (34.67 ± 1.53) compared to the “w/o RL” variant (36.33 ± 3.06). Similar consistent improvements are observed across 10n, 30n, and 50n nonlinear datasets, as well as the 100n linear dataset. These findings indicate that the stochastic exploration provided by the RL policy gradient explicitly helps the model navigate the complex discrete structural space, actively pruning false-positive edges and improving overall graph accuracy.Table 5SHD comparison under linear settingsMethod10n30n50n100nDAG-VAERL 1.0 ± 1.73 1.0 ± 1.0 4.0 ± 3.61 12.0 ± 13.08w/o RL 1.0 ± 1.73 1.0 ± 1.0 4.0 ± 3.61 13.33 ± 13.05w/o DAG 11.67 ± 2.08 29.0 ± 4.36 52.33 ± 6.35 106.0 ± 3.0Table 6F1 score comparison under linear settingsMethod10n30n50n100nDAG-VAERL 0.96 ± 0.08 0.98 ± 0.02 0.96 ± 0.04 0.94 ± 0.06w/o RL 0.96 ± 0.08 0.98 ± 0.02 0.96 ± 0.04 0.93 ± 0.06w/o DAG 0.57 ± 0.03 0.57 ± 0.02 0.58 ± 0.05 0.58 ± 0.02Table 7SHD comparison under nonlinear settingsMethod10n30n50n100nDAG-VAERL 6.33 ± 4.04 9.33 ± 0.58 19.0 ± 7.94 34.67 ± 1.53w/o RL 7.67 ± 3.51 10.0 ± 3.61 20.33 ± 4.73 36.33 ± 3.06w/o DAG 8.33 ± 5.03 16.33 ± 1.53 32.67 ± 7.23 60.67 ± 3.21Table 8F1 score comparison under nonlinear settingsMethod10n30n50n100nDAG-VAERL 0.64 ± 0.15 0.80 ± 0.04 0.80 ± 0.09 0.80 ± 0.03w/o RL 0.57 ± 0.10 0.78 ± 0.07 0.78 ± 0.05 0.79 ± 0.02w/o DAG 0.56 ± 0.24 0.69 ± 0.02 0.67 ± 0.06 0.68 ± 0.04 Table 5SHD comparison under linear settingsMethod10n30n50n100nDAG-VAERL 1.0 ± 1.73 1.0 ± 1.0 4.0 ± 3.61 12.0 ± 13.08w/o RL 1.0 ± 1.73 1.0 ± 1.0 4.0 ± 3.61 13.33 ± 13.05w/o DAG 11.67 ± 2.08 29.0 ± 4.36 52.33 ± 6.35 106.0 ± 3.0 SHD comparison under linear settings Table 6F1 score comparison under linear settingsMethod10n30n50n100nDAG-VAERL 0.96 ± 0.08 0.98 ± 0.02 0.96 ± 0.04 0.94 ± 0.06w/o RL 0.96 ± 0.08 0.98 ± 0.02 0.96 ± 0.04 0.93 ± 0.06w/o DAG 0.57 ± 0.03 0.57 ± 0.02 0.58 ± 0.05 0.58 ± 0.02 F1 score comparison under linear settings Table 7SHD comparison under nonlinear settingsMethod10n30n50n100nDAG-VAERL 6.33 ± 4.04 9.33 ± 0.58 19.0 ± 7.94 34.67 ± 1.53w/o RL 7.67 ± 3.51 10.0 ± 3.61 20.33 ± 4.73 36.33 ± 3.06w/o DAG 8.33 ± 5.03 16.33 ± 1.53 32.67 ± 7.23 60.67 ± 3.21 SHD comparison under nonlinear settings Table 8F1 score comparison under nonlinear settingsMethod10n30n50n100nDAG-VAERL 0.64 ± 0.15 0.80 ± 0.04 0.80 ± 0.09 0.80 ± 0.03w/o RL 0.57 ± 0.10 0.78 ± 0.07 0.78 ± 0.05 0.79 ± 0.02w/o DAG 0.56 ± 0.24 0.69 ± 0.02 0.67 ± 0.06 0.68 ± 0.04 F1 score comparison under nonlinear settings Applications We consider a bioinformatics data set for discovering a causal network based on expression levels of proteins, genes, and lncRNA. Gene causality plays a vital role in understanding the mechanisms of human biology and diseases. Here, the mechanisms of Alzheimer’s disease (AD) at the gene level will be detected by our method. We consider one of the largest AD transcriptome sequencing datasets, recording the gene expression levels of over 36,000 genes across more than 20,000 samples isolated from 24 individuals from the Religious Orders Study and Memory and Aging Project (ROSMAP) study [17]; the data is publicly available through the Synapse software platform under the ID: syn3219045. Each sample consists of a single nucleus isolated and profiled using the DroNc-seq protocol, a droplet-based technology designed for high-throughput, massively parallel single-nucleus RNA sequencing (snRNA-seq). Data preprocessing and gene selection To construct a biologically meaningful and computationally tractable causal graph, we applied a rigorous preprocessing and gene selection pipeline to the raw ROSMAP single-nucleus RNA-sequencing (snRNA-seq) dataset. First, we performed standard quality control to remove low-quality data: cells expressing fewer than 200 genes or exhibiting a high mitochondrial gene ratio ( > 5%) were excluded, and genes expressed in fewer than 3 cells were discarded. The remaining count matrix was then log-normalized to account for sequencing depth variations. Given the high dimensionality of the transcriptome, learning a global DAG across all genes is highly susceptible to false positives. Therefore, we performed a multi-step filtering process to select highly expressed and biologically relevant nodes. Specifically, we first filtered for highly expressed genes by retaining only those with an average log-normalized expression greater than 0.5 across all cells. Next, to capture the pathogenic mechanisms of Alzheimer’s disease, we performed differential expression analysis between AD patients and healthy controls, retaining genes with statistical significance (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm{FDR} < 0.05$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\log_2 \mathrm{FC}| > 0.5$$\end{document}). Finally, to focus on the core regulatory networks, we intersected these highly expressed and differentially expressed candidates with a curated list of established AD-associated protein-coding genes, transcription factors, and lncRNAs. In the current DAG-VAERL framework, lncRNAs, transcription factors, and protein-coding genes are represented uniformly as expression-based nodes in the learned graph. This design allows the model to infer directed dependencies among all selected molecular features using the same optimization framework. However, their biological identities are retained during downstream interpretation. In particular, lncRNAs are included as candidate regulatory nodes because one motivation of this study is to investigate potential lncRNA-mediated regulatory relationships in Alzheimer’s disease. We note that the current model is not node-type-aware and does not impose different priors, edge constraints, or regulatory roles for lncRNAs, transcription factors, and protein-coding genes. Causal assumptions for observational data The ROSMAP dataset consists of purely observational transcriptomic data. To theoretically justify the interpretation of the learned Directed Acyclic Graphs (DAGs) as causal gene regulatory networks, our causal discovery framework relies on three standard causal assumptions:Causal Markov Condition: We assume that the expression of a gene is independent of its non-descendants given its direct parents in the causal DAG.Faithfulness: We assume that all conditional independencies observed in the empirical transcriptomic data perfectly correspond to the structural d-separations within the true causal graph.Causal Sufficiency: We assume there are no unmeasured latent confounders that jointly regulate any two or more genes within our selected subset. Causal Markov Condition: We assume that the expression of a gene is independent of its non-descendants given its direct parents in the causal DAG. Causal Markov Condition: We assume that the expression of a gene is independent of its non-descendants given its direct parents in the causal DAG. Faithfulness: We assume that all conditional independencies observed in the empirical transcriptomic data perfectly correspond to the structural d-separations within the true causal graph. Faithfulness: We assume that all conditional independencies observed in the empirical transcriptomic data perfectly correspond to the structural d-separations within the true causal graph. Causal Sufficiency: We assume there are no unmeasured latent confounders that jointly regulate any two or more genes within our selected subset. Causal Sufficiency: We assume there are no unmeasured latent confounders that jointly regulate any two or more genes within our selected subset. We acknowledge that causal sufficiency is a strong assumption for scRNA-seq data. In real biological systems, unmeasured factors such as upstream regulators, chromatin accessibility, protein-level regulation, batch effects, cell-state heterogeneity, and other latent biological processes may influence multiple genes simultaneously. Therefore, the inferred DAGs should be interpreted as putative regulatory structures under standard observational causal discovery assumptions, rather than as definitive causal graphs. This clarification is important for defining the scope and limitations of applying DAG-VAERL to observational transcriptomic data. Results on alzheimer’s disease network To validate our result, we constructed the gene causal network to reveal the mechanisms of Alzheimer’s disease at a single-cell level. After acquiring the gene causal network, we separate the whole gene causal DAG into different gene sink graphs. In our result, the most AD-relevant gene sink graphs built by our method contain multiple AD-related genes and some lncRNAs that affect other genes or proteins. Figure 6 illustrates the complete gene causal network, while Fig. 7 presents the gene sink graph specifically for KAZN. From Fig. 7, it is evident that nearly all arrows point toward KAZN, identifying it as the ultimate sink node in the network. This observation aligns with existing research, as several studies have highlighted the association between KAZN and Alzheimer’s disease [18–20], further supporting its potential role in the disease’s underlying mechanisms.Fig. 6Whole gene causal networkFig. 7Gene sink graph of KAZN Whole gene causal network Gene sink graph of KAZN To provide exploratory biological context for the inferred causal relationships, we performed a downstream pathway enrichment analysis on the sub-network connected to CAMK2N1. We extracted the immediate neighbors of CAMK2N1 from the learned DAG and conducted a hypergeometric test against the Reactome Pathways database. As shown in Table 9, the analysis suggested enrichment in Notch-related pathways, with the strongest signal observed for the Notch signaling pathway (p = 0.037), and HES5 was identified as a key downstream effector.Table 9Reactome pathway enrichment analysis of the CAMK2N1-associated sub-networkPathway NameAdjustedP-valueSignaling by NOTCH1 PEST Domain Mutants0.037Signaling by NOTCH10.040Signaling by NOTCH40.040 Table 9Reactome pathway enrichment analysis of the CAMK2N1-associated sub-networkPathway NameAdjustedP-valueSignaling by NOTCH1 PEST Domain Mutants0.037Signaling by NOTCH10.040Signaling by NOTCH40.040 Reactome pathway enrichment analysis of the CAMK2N1-associated sub-network We interpret this result as supportive biological evidence rather than definitive causal validation, because the true causal gene-regulatory graph for the Alzheimer’s disease dataset is unknown. A possible connection between CAMK2N1 and the Notch signaling pathway may represent a biologically plausible regulatory axis in Alzheimer’s disease pathogenesis. CAMK2N1 is a potent endogenous inhibitor that specifically targets CaMKII. CaMKII, a master kinase in neurons, is known to facilitate the activation and nuclear translocation of the Notch intracellular domain (NICD). Consequently, the over-expression or dysregulation of CAMK2N1 may act as a “molecular brake” that indirectly modulates the intensity of Notch-driven transcriptional responses, such as the expression of HES5 [21]. This potential interplay is relevant to Alzheimer’s disease pathogenesis, as both Notch and the amyloid precursor protein (APP) serve as substrates for the γ-secretase complex [22]. Further validation using additional pathway databases, external regulatory resources, or perturbation-based experiments will be needed to confirm this inferred regulatory relationship. We have observed an intriguing phenomenon involving certain pairwise gene interactions, such as the pair of STAR and MEF2A. The representative pairwise causal relationships are shown in Fig. 8. Current literature indicates that that both genes are implicated in Alzheimer’s disease (AD) through distinct yet potentially interconnected mechanisms. STAR (Steroidogenic Acute Regulatory Protein) is primarily involved in steroid hormone synthesis [23], and its dysregulation may contribute to AD pathology by altering neuroprotective steroid levels, such as estrogen, which are critical for neuronal survival and synaptic plasticity. On the other hand, MEF2A (Myocyte Enhancer Factor 2A) is a transcription factor that regulates genes associated with neuronal development, synaptic function, and cell survival [24]. In AD, impaired MEF2A activity has been linked to synaptic dysfunction and neuronal loss.Fig. 8Pairwise causal genes Pairwise causal genes A possible connection between STAR and MEF2A can be hypothesized through several mechanisms. First, in the context of neuroprotection and synaptic plasticity, STAR-mediated regulation of steroid hormones [23], such as estrogen, may indirectly influence MEF2A activity, as estrogen has been shown to activate MEF2A [25], thereby promoting synaptic plasticity and neuronal survival. Second, both genes may play roles in inflammation and oxidative stress, key features of AD pathology. STAR could modulate inflammatory responses through steroid hormone signaling, while MEF2A might mitigate inflammation and oxidative damage via its transcriptional regulatory functions [26]. Lastly, regarding gene expression regulation, MEF2A, as a transcription factor, could directly or indirectly regulate STAR expression, or both genes might be co-regulated by shared upstream signaling pathways, such as the cAMP/PKA pathway [27]. These potential interactions suggest a complex interplay between STAR and MEF2A in AD, warranting further investigation to elucidate their combined impact on disease progression. In addition to the STAR–MEF2A interaction, we also identified an association between MALAT1 and NFKB1 in our inferred network. MALAT1 is a well-characterized long non-coding RNA (lncRNA) with established roles in transcriptional and inflammatory regulation. Previous studies have shown that MALAT1 can modulate NF-κB signaling activity, for example by influencing NF-κB activation and downstream inflammatory gene expression [28]. The recovery of this biologically supported MALAT1–NFKB1 relationship in our model further highlights the potential contribution of lncRNAs to disease-related regulatory programs and supports the capacity of our framework to capture functionally meaningful lncRNA-mediated associations. Discussion In this study, we proposed the DAG-VAERL framework, which integrates Graph Neural Networks (GNNs) and Reinforcement Learning (RL) within a Variational Autoencoder (VAE) architecture to learn the structure of Directed Acyclic Graphs (DAGs). Our experimental results demonstrate that DAG-VAERL achieves strong and robust performance compared with established DAG learning algorithms—such as DAG-GNN [8], CASPER [11], NOTEARS [7], and the PC algorithm [2]—on both synthetic benchmarks and real-world datasets. The model shows particular advantages when processing data characterized by larger graph structures and complex nonlinear relationships. It is important to note that DAG-VAERL does not outperform DAG-GNN in all settings. In the very small 10-node benchmarks, DAG-GNN achieves near-perfect or perfect structural recovery, whereas DAG-VAERL shows slightly lower performance, particularly in the nonlinear setting. This is expected because the search space is relatively small at this scale, and the DAG-GNN backbone is already sufficient to recover the underlying graph accurately. In such cases, the additional RL-based exploration may introduce variance whose cost can exceed its benefit. Moreover, the VAE backbone combined with multi-head attention has a higher model capacity, which can be over-parameterized for a 10-node graph. The primary design goal of DAG-VAERL is therefore not to replace simpler continuous-optimization methods on very small graphs, but to improve robustness and structural exploration in medium- and larger-scale settings. As the number of nodes increases from 30 to 100, DAG-VAERL maintains F1 scores around 0.9 with stable training behavior, indicating that its advantage becomes more apparent as the graph structure and search space become more complex. This makes DAG-VAERL particularly suitable for nonlinear and high-dimensional biological data, where expressive representation learning and controlled structural exploration are both important. Specifically, DAG-VAERL effectively recovers the underlying DAG structure from nonlinear data in medium- and large-scale settings, even as the network scales to 100 nodes. It surpasses many existing traditional methods across various evaluation metrics, including Structural Hamming Distance (SHD), precision, and recall. This indicates that DAG-VAERL can capture complex nonlinear dependencies and maintain strong stability in higher-dimensional spaces. Compared to constraint-based PC algorithms [2] and score-based NOTEARS [7] methods, DAG-VAERL shows significant robustness across multiple tasks. A core innovation of DAG-VAERL lies in its reinforcement learning strategy, which dynamically adjusts the graph structure during training. Traditional continuous methods often suffer from local optima in highly non-convex landscapes. In contrast, our RL module utilizes a carefully designed reward mechanism to guide the agent in actively selecting true causal edges and penalizing false positives. This dynamic optimization of the discrete adjacency matrix significantly enhances the model’s structural learning capability, overcoming the limitations of purely continuous optimization and rigid conditional independence tests. In terms of architecture, the encoder of DAG-VAERL employs a multi-head self-attention mechanism, enabling the model to capture long-range structural dependencies and enhancing its representational power. This GNN-based attention architecture naturally accommodates graph-structured data, demonstrating broad application potential in complex causal inference tasks. While DAG-VAERL achieves strong structural accuracy in medium- and large-scale settings, it is important to acknowledge its computational overhead. We deliberately trade off computational speed to achieve higher precision and capture complex nonlinear cascades. Consistent with standard continuous deep learning baselines, our current evaluation is practically bounded to graphs of roughly 100 to 200 nodes due to the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{O}(d^3)$$\end{document} computational complexity inherent in evaluating the matrix trace exponential for the acyclicity constraint. Scaling exact global DAG learning to massive networks containing thousands of nodes remains a field-wide challenge. We anticipate mitigating this limitation in future work by exploring relaxed acyclicity penalties, localized causal discovery algorithms, and leveraging stronger hardware acceleration. Conclusion In this work, we introduced DAG-VAERL, a novel framework combining Graph Neural Networks (GNNs) and Reinforcement Learning (RL) for the challenging task of Directed Acyclic Graph (DAG) structure learning. Unlike traditional methods, DAG-VAERL directly addresses the bottlenecks of handling complex nonlinear dependencies by embedding the causal search process within a Variational Autoencoder (VAE) latent space. Our extensive experiments on both synthetic datasets and a real-world Alzheimer’s disease single-cell dataset demonstrate that DAG-VAERL consistently outperforms existing algorithms, with particular success in deciphering intricate nonlinearity and high dimensionality. The integration of Reinforcement Learning proved to be crucial for navigating the discrete DAG space. By enabling dynamic, reward-driven updates to the adjacency matrix during training, the RL module significantly improves the accuracy and robustness of the learned graph, overcoming the instability associated with manually designed scoring functions. Furthermore, the incorporation of a multi-head self-attention mechanism in the encoder empowers the model to capture long-range dependencies, further strengthening the representation learning process. Overall, these results highlight the effectiveness of DAG-VAERL in causal inference and structural graph analysis, demonstrating its immense potential for broader applications in domains requiring precise causal structure learning. The model’s ability to accurately distill causal hierarchies from complex, high-dimensional observational data establishes it as a robust and promising tool for future research in real-world causal discovery.