A BGe score for tied-covariance mixtures of Gaussian Bayesian networks

2025

Read A BGe score for tied-covariance mixtures of Gaussian Bayesian networks (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07050.txt.

Introduction

Marco Grzegorczyk studies mixtures of Gaussian Bayesian networks, probabilistic models that blend several network structures to capture heterogeneous data. Earlier work let every mixture part carry its own covariance matrix. Grzegorczyk instead proposes tied covariance, where all parts share one common covariance matrix, a simpler and more constrained model that can resist overfitting.

His main contribution derives the model's marginal likelihood, the BGe score used to compare structures, and shows it remains available in closed analytic form. Unlike the full-covariance case, however, the tied score no longer splits into separate part-certain factors. For inference he implements Markov Chain Monte Carlo schemes that combine structure search with a fast Gibbs sampler for the mixture assignments. Compares tied against full covariance on simulated and benchmark data.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07050

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A BGe score for tied-covariance mixtures of Gaussian Bayesian networks Abstract Mixtures of Gaussian Bayesian networks have previously been studied under full-covariance assumptions, where each mixture component has its own covariance matrix. We propose a mixture model with tied-covariance, in which all components share a common covariance matrix. Our main contribution is the derivation of its marginal likelihood, which remains analytic. Unlike in the full-covariance case, however, the marginal likelihood no longer factorizes into component-specific terms. We refer to the new likelihood as the BGe scoring metric for tied-covariance mixtures of Gaussian Bayesian networks. For model inference, we implement MCMC schemes combining structure MCMC with a fast Gibbs sampler for mixtures, and we empirically compare the tied- and full-covariance mixtures of Gaussian Bayesian networks on simulated and benchmark data. Marco Grzegorczyk Bernoulli Institute (BI) Rijksuniversiteit Groningen E-mail: [email protected] Keywords Gaussian Bayesian networks, mixture model, tied-covariance, Markov chain Monte Carlo 1 Introduction Bayesian networks (BNs) are popular graphical models for representing conditional (in-) dependencies among random variables using directed acyclic graphs (DAGs). Each node in a DAG corresponds to a variable, and directed edges encode probabilistic relationships. This structure allows joint distributions with many conditional independencies to be expressed compactly. BNs are widely used for learning under uncertainty, making them important tools in statistics, machine learning, and artificial intelligence. Thorough introductions to Bayesian networks are available in standard textbooks ( 33 ; 30 ; 25 ; 36 ) . Structure learning algorithms aim to infer DAGs from data, but since the search space grows super-exponentially with the number of nodes ( 4 ) , the problem is computationally challenging. For efficient structure learning algorithms, see 27 ; 38 ; 37 ; 40 ; 23 . In the case of sparse data, there is typically considerable uncertainty about the DAG. This uncertainty can be addressed by Bayesian structure learning algorithms, by sampling DAGs from the posterior distribution and averaging over the sampled DAGs. This approach is commonly referred to as Bayesian model averaging ( 7 ) . For Bayesian BN structure learning algorithms, see 29 ; 15 ; 11 or 17 . Beyond the standard BN structure learning literature, tailored approaches have been developed for incomplete ( 12 ; 34 ; 2 ) , ordinal ( 28 ; 8 ; 35 ) , and hybrid data ( 5 ; 40 ) . We have recently contributed to this line of work by proposing Bayesian counterparts for these important special cases, addressing missing ( 18 ) , ordinal ( 19 ) , and hybrid ( 20 ) data. In the present paper, we extend this trilogy ( 18 ; 19 ; 20 ) by introducing a Bayesian approach for tied-covariance mixtures of Gaussian Bayesian networks. A mixture model assumes that the data stem from several subpopulations, each represented by its own distribution. Mixture models are used in many frameworks, and we are not the first to propose mixture Bayesian networks: both non-Bayesian ( 24 ) and Bayesian ( 16 ) formulations have been introduced, each assuming that all subpopulations share a common DAG but differ in their network parameters. Because they use component-specific covariance matrices, these earlier models are full-covariance mixtures of Gaussian Bayesian networks, which we refer to as M1. The Bayesian M1 model from 16 employs the structure MCMC sampler ( 29 ; 15 ) to infer the DAGs and the allocation sampler ( 32 ) to assign the data to K mixture components. Each mixture component k=1,\ldots,K has a weight \pi_{k}\in[0,1] , a mean vector \mbox{{\boldmath$\mu$}}_{k} , and a covariance matrix \mbox{{\boldmath$\Sigma$}}_{k} . The resulting density is p({\bf x}|\mbox{DAG},\mathcal{M}_{1})=\sum_{k=1}^{K}\pi_{k}\cdot\mathcal{N}_{n}({\bf x}|\mbox{{\boldmath$\mu$}}_{k},\mbox{{\boldmath$\Sigma$}}_{k}), where \mathcal{N}_{n}({\bf x}|\mbox{{\boldmath$\mu$}},\mbox{{\boldmath$\Sigma$}}) denotes the density of the n -variate normal distribution \mathcal{N}_{n}(\mbox{{\boldmath$\mu$}},\mbox{{\boldmath$\Sigma$}}) at {\bf x} , and each \mbox{{\boldmath$\Sigma$}}_{k} must be consistent with the DAG, that is, each \mbox{{\boldmath$\Sigma$}}_{k} must encode the conditional independence relations implied by the DAG. For a dataset \mathcal{D}=\{{\bf x}_{1},\ldots,{\bf x}_{m}\} with given class labels and under suitable prior assumptions, 1 1 1 In each component k , an independent Normal–Wishart prior is used ( 13 ; 14 ; 26 ) . the marginal likelihood of the mixture BN model factorizes into the product of the component-specific marginal likelihoods: p(\mathcal{D}|\mbox{DAG},\mathcal{M}_{1})=\prod_{k=1}^{K}p(\mathcal{D}_{k}|\mbox{DAG}), where \mathcal{D}_{k} denotes the subset of observations with class label k , and each p(\mathcal{D}_{k}|\mbox{DAG}) can be computed analytically. In this paper, we introduce the tied-covariance counterpart of the M1 model, which we refer to as the M2 mixture model. In the M2 model the K components share a common covariance matrix \Sigma : p({\bf x}|\mbox{DAG},\mathcal{M}_{2})=\sum_{k=1}^{K}\pi_{k}\cdot\mathcal{N}_{n}({\bf x}|\mbox{{\boldmath$\mu$}}_{k},\mbox{{\boldmath$\Sigma$}}). The main contribution of this paper is the derivation of the marginal likelihood p(\mathcal{D}|\text{DAG},\mathcal{M}_{2}) for the tied-covariance mixture model (M2). We show that it admits an analytic solution, but unlike in the M1 case, it no longer decomposes into K local marginal likelihoods. We refer to the new marginal likelihood as the BGe scoring metric for tied-covariance mixtures of Gaussian Bayesian networks. We implement both types of mixture models and compare their performances on simulated and benchmark data. Our Markov chain Monte Carlo (MCMC) sampling algorithm combines structure MCMC sampling of DAGs ( 29 ; 15 ) with a recently proposed fast Gibbs sampler for mixture models ( 31 ) . 2 Statistical methods 2.1 Bayesian networks Bayesian networks (BNs) are graphical models that use directed acyclic graphs (DAGs) to represent conditional independencies among a set of random variables, which we denote by X_{1},\ldots,X_{n} . The variables correspond to the n nodes of a DAG, and directed edges between nodes encode direct conditional dependencies. Node X_{j} is called a parent of X_{i} if there is an edge X_{j}\rightarrow X_{i} , and it is called an ancestor of X_{i} if X_{i} can be reached from X_{j} by following a sequence of one or more directed edges along their orientations, symbolically X_{j}\rightarrow\ldots\rightarrow X_{i} . By definition, a node cannot be its own ancestor, so cyclic paths of directed edges, such as X_{i}\rightarrow\ldots\rightarrow X_{i} , are excluded. The set of parent nodes of X_{i} is denoted by \mbox{{\boldmath$\Pi$}}_{i} . A DAG can be identified with its n parent sets \mbox{{\boldmath$\Pi$}}_{1},\ldots,\mbox{{\boldmath$\Pi$}}_{n} . Given \mathcal{G} , i.e. the conditional independence relations implied by \mathcal{G} , the joint density factorizes as p(X_{1},\ldots,X_{n}|\mathcal{G})=\prod\limits_{i=1}^{n}p(X_{i}|\mbox{{\boldmath$\Pi$}}_{i}). (1) Two DAGs, say \mathcal{G}_{\star} and \mathcal{G}_{\bullet} , that imply the same conditional independencies are called equivalent, and they yield the same factorization: p(X_{1},\ldots,X_{n}|\mathcal{G}_{\star})=\prod\limits_{i=1}^{n}p(X_{i}|\mbox{{\boldmath$\Pi$}}^{\star}_{i})=\prod\limits_{i=1}^{n}p(X_{i}|\mbox{{\boldmath$\Pi$}}^{\bullet}_{i})=p(X_{1},\ldots,X_{n}|\mathcal{G}_{\bullet}), where \mbox{{\boldmath$\Pi$}}^{\star}_{i} and \mbox{{\boldmath$\Pi$}}^{\bullet}_{i} denote the parent sets of X_{i} in \mathcal{G}_{\star} and \mathcal{G}_{\bullet} , respectively. Equivalent DAGs belong to the same equivalence class. 6 shows that DAG equivalence classes can be represented by completed partially directed acyclic graphs (CPDAGs), and that two DAGs \mathcal{G}_{\star} and \mathcal{G}_{\bullet} are equivalent if and only if they share the same skeleton and the same v-structures. The skeleton of a DAG is obtained by replacing each directed edge with an undirected edge. A v-structure is a configuration in which a node X_{i} has two parent nodes with no edge between them, for example X_{k}\rightarrow X_{i}\leftarrow X_{l} with no edge between X_{k} and X_{l} . CPDAGs contain a mixture of directed and undirected edges. A directed edge X_{j}\rightarrow X_{i} in a CPDAG indicates that all DAGs in the equivalence class share this directed edge. An undirected edge X_{j}-X_{i} indicates that all DAGs in

Read the complete text