Micro-DeMix: a mixture beta-multinomial model for investigating the heterogeneity of the stool microbiome compositions Authors: Ruoqian Liu, Yue Wang, Dan Cheng Published: 2024 ABSTRACT AbstractMotivationExtensive research has uncovered the critical role of the human gut microbiome in various aspects of health, including metabolism, nutrition, physiology, and immune function. Fecal microbiota is often used as a proxy for understanding the gut microbiome, but it represents an aggregate view, overlooking spatial variations across different gastrointestinal (GI) locations. Emerging studies with spatial microbiome data collected from specific GI regions offer a unique opportunity to better understand the spatial composition of the stool microbiome.ResultsWe introduce Micro-DeMix, a mixture beta-multinomial model that deconvolutes the fecal microbiome at the compositional level by integrating stool samples with spatial microbiome data. Micro-DeMix facilitates the comparison of microbial compositions across different GI regions within the stool microbiome through a hypothesis-testing framework. We demonstrate the effectiveness and efficiency of Micro-DeMix using multiple simulated datasets and the inflammatory bowel disease data from the NIH Integrative Human Microbiome Project.Availability and implementationThe R package is available at https://github.com/liuruoqian/MicroDemix. 1 Introduction The human gut microbiome and its role in host health have been the subject of extensive research, establishing its involvement in human metabolism (Devaraj et al. 2013), nutrition (Zhang 2022), physiology (Andoh 2016), and immune function (Wu and Wu 2012, Bull and Plummer 2014). Recent studies have also demonstrated the association between the gut microbiota and the emergence of obesity (Aoun et al. 2020), metabolic syndrome and the onset of type 2 diabetes (Devaraj et al. 2013, Iatcu et al. 2021). Moreover, altered composition and function of the gut microbiota has been found associated with chronic diseases (Pellanda et al. 2021) ranging from gastrointestinal (GI) inflammatory (Shan et al. 2022) and metabolic conditions to neurological (Cryan et al. 2020), cardiovascular (Tang et al. 2017), and respiratory illnesses (Durack and Lynch 2019, Chunxi et al. 2020). Most scientific findings regarding the gut microbiome have been derived from stool samples. However, it is increasingly recognized that the stool microbiome offers only an aggregate view, overlooking the spatial heterogeneity of the microbiome across different GI locations (Ahn et al. 2023, Levitan et al. 2023). Recent studies have revealed significant variations in microbial composition and function along distinct segments of the GI tract (Leite et al. 2020). For instance, the small intestine is dominated by Lactobacillaceae and Enterobacteriaceae, whereas the colon harbors species from families such as Bacteroidaceae, Prevotellaceae, Rikenellaceae, Lachnospiraceae, and Ruminococcaceae (Donaldson et al. 2016). Furthermore, spatial metagenomics applied to the mouse colonic microbiome has revealed a heterogeneous distribution of taxa throughout the gut (Sheth et al. 2019). Thus, understanding the spatial composition of the stool microbiome is essential for enhancing the biological relevance of stool-based microbiome analyses, allowing for more accurate interpretations of its role in health and disease. Although spatial microbiome data from specific GI locations remain relatively rare due to the invasive nature of sample collection, they offer a unique opportunity to deconvolute stool samples, thereby providing a deeper understanding of the gut microbiome’s spatial heterogeneity. We develop Micro-DeMix, a novel statistical model designed to deconvolute the fecal microbiome into specific GI locations by integrating spatial microbiome data. The development of Micro-DeMix was motivated by data from the inflammatory bowel disease (IBD) Multiomics Database (IBDMDB) project, part of the NIH Integrative Human Microbiome Project (iHMP). This project followed 132 individuals and collected 1785 stool samples and 651 intestinal biopsies over time (Integrative HMP et al. 2019). The integrated nature of this dataset, combining microbial profiles from both stool samples and distinct GI locations, highlighted the need for a tool like Micro-DeMix to effectively analyze and integrate these diverse sources of microbiome data. Micro-DeMix is a mixture beta-multinomial model developed to deconvolute the fecal microbiome into specific GI locations. The multinomial component effectively handles the compositional nature of microbiome data by jointly modeling microbial proportions, while the beta component adjusts for sample heterogeneity, incorporating key demographic and clinical variables that may influence the composition of the fecal microbiome. We introduce two estimation procedures for the parameters in the Micro-DeMix model. The first procedure directly maximizes the likelihood of the observed microbial counts, offering computational efficiency. This approach is also equipped with an efficient hypothesis-testing framework for comparing microbial compositions from different GI regions within the mixed stool sample based on the asymptotic null distribution. The second procedure is based on the Expectation-Maximization (EM) algorithm, which provides greater numerical robustness than the first method. It is particularly useful for scenarios with extremely rare microbes, where the first procedure may encounter limitations, although it is computationally more intensive. This EM approach includes a permutation test to identify differentially abundant microbial groups across GI locations. Unlike conventional differential abundance (DA) tools, which compare microbial abundances between predefined groups (e.g. disease versus healthy or across body sites), Micro-DeMix is specifically designed to detect compositional changes within a mixed population (e.g. the stool sample), where the groups are not prespecified and must be statistically estimated. This enables Micro-DeMix to uncover group differences and spatial heterogeneity within the stool microbiome by integrating spatial microbiome data. Traditional DA tools, which rely on predefined group distinctions, are less suited to detect these types of within-sample variations. We demonstrate the effectiveness of Micro-DeMix through extensive simulation studies and an application to IBD cohorts, where we integrate stool microbiome data with rectum microbiome data. The rest of the paper is organized as follows. In Section 2, we describe the Micro-DeMix model, develop two maximum-likelihood-based estimation procedures, and propose hypothesis-testing frameworks for detecting differentially abundant microbes between GI locations. In Section 3, we conduct simulation studies to demonstrate the effectiveness of Micro-DeMix in terms of the estimation accuracy and the power of the hypothesis testing method. In Section 4, we apply our method to the IBD microbiome data from iHMP to elucidate the composition of fecal microbiome in IBD patients. We close with a discussion of our method in Section 5. 2 Materials and methods 2.1 The Micro-DeMix model In this section, we present a beta-multinomial framework for modeling microbial abundance in stool samples. Our model considers a multivariate extension of the beta-binomial model proposed in a recent research (Martin et al. 2020) which models individual taxa separately. As shown later, while this multivariate extension allows for joint modeling of multiple microbes, it also brings several new statistical and computational challenges. For ease of presentation and biological relevance, we assume the reference spatial microbiome data are collected at the rectum. However, the methodology proposed below can seamlessly extend to other segments of the GI tract, including the ileum, jejunum, colon, and beyond, given the data availability. Let yjg(r) denotes the observed absolute abundance for taxon g in the jth subject of the reference rectum microbiome data for j=1,…,S(r). Let yig denotes the observed absolute abundance for taxon g in the ith stool samples for g=1,…,G and i=1,…,S. For all subjects, we also collect covariate information xj(r) and xi for j=1,…,S(r) and i=1,…,S. We model the stool microbial abundance yi=(yi1,…,yiG)⊺ using a multinomial model yi|pi∼Multinomial(Ni,pi), where Ni=∑g=1Gyig and pi=(pi1,…,piG)⊺ is the underlying true microbial proportions. We are particularly interested in the spatial composition of the stool microbiome. Acknowledging that the stool microbiome consists of microbes from rectum and other GI locations (mostly, the small intestine), we consider (1)pig=πipg(r)+(1−πi)pg(o);here, πi∈(0,1) denotes the latent proportion of the rectal microbiome in the ith stool sample; pg(r) and pg(o) denotes the proportion of taxon g in the rectum and other GI locations, respectively. To allow πi to vary across individuals, we assume πi∼Beta(a1,i,a2,i). Hence, we write the joint density function of (yig,πi) as (2)f(yig,πi|pg(r),pg(o),a1,i,a2,i)=Ni!yi1!…yiG!∏g=1G{πipg(r)+(1−πi)pg(o)}yigπia1,i−1(1−πi)a2,i−1B(a1,i,a2,i), where B(a1,i, a2,i) = Γ(a1,i)Γ(a2,i)/Γ(a1,i + a2,i) To capture the association between πi and the subjects’ characteristics, we further link each πi with covariates xi through a beta-regression model. With additional reparameterization (3)μi=a1,ia1,i+a2,i,(4)ϕi=a1,i+a2,i,we consider (5)log(μi1−μi)=β0+xi⊺β,(6) log(ϕi)=γ0+xi⊺γ. Some calculations yield that μi is the expected proportion of the rectum microbiome in the ith stool sample, i.e. μi=E(πi), and Var(πi)=μi(1−μi)/(1+ϕi), where ϕi is known as the precision parameter. Thus, our first goal is to estimate each pg(r) and pg(o) to understand the spatial heterogeneity of the stool microbiome. Then, we aim to identify microbial groups that are differentially abundant between the rectum and other GI locations with the stool sample. Below we will present two procedures for these two analyses, each with its own strengths for different types of applications.Remark 1.Under the proposed model, we derive the mean and variance of yig: E(yig)=Ni[pg(r)a1,ia1,i+a2,i+pg(o)(1−a1,ia1,i+a2,i)]Var(yig)=Ni{a1,ia1,i+a2,i(pg(r)−2pg(r)pg(o)−pg(o)+2(pg(o))2)+(a1,i2(a1,i+a2,i)2+a1,ia2,i(a1,i+a2,i)2(a1,i+a2,i+1))(−(pg(r))2+2pg(r)pg(o)−(pg(o))2)+pg(o)−(pg(o))2}+Ni2(pg(r)−pg(o))2a1,ia2,i(a1,i+a2,i)2(a1,i+a2,i+1).See derivations in the Supplementary Material. Under the proposed model, we derive the mean and variance of yig: E(yig)=Ni[pg(r)a1,ia1,i+a2,i+pg(o)(1−a1,ia1,i+a2,i)]Var(yig)=Ni{a1,ia1,i+a2,i(pg(r)−2pg(r)pg(o)−pg(o)+2(pg(o))2)+(a1,i2(a1,i+a2,i)2+a1,ia2,i(a1,i+a2,i)2(a1,i+a2,i+1))(−(pg(r))2+2pg(r)pg(o)−(pg(o))2)+pg(o)−(pg(o))2}+Ni2(pg(r)−pg(o))2a1,ia2,i(a1,i+a2,i)2(a1,i+a2,i+1). See derivations in the Supplementary Material. 2.2 Micro-DeMix-1 2.2.1 Estimation The joint likelihood in Equation (2) involves πi, which is unobserved. Thus, we calculate the log-likelihood of observing {yig:i=1,…,S;g=1,…,G} in stool samples: (7)log L(θ)=∑i=1S log ∫01f(yig,πi|θ)dπi,where θ=(p(r),p(o),β0,β,γ0,γ)⊺. We estimate θ in two steps. We first estimate pg(r) using the reference rectum microbiome data for g=1,…,G. Recall that yjg(r) denotes the observed absolute abundance for taxon g and sample j, j=1,…,S(r), collected from rectum. Let Nj(r) denotes the sequencing depth for the jth sample from rectum. We assume the absolute abundance to follow a multinomial distribution (8)yj(r)∼Multinomial(Nj(r),p(r)). Based on this model, we estimate pg(r) with the sample proportions (9)p^g(r)=∑jyjg(r)∑jNj(r). After replacing pg(r) in Equation (7) by p^g(r), we estimate the remaining parameters using an iterative procedure with the main steps given in Algorithm 1.Algorithm 1.Estimating {pg(o),β0,β,γ0,γ} for Micro-DeMixRequire:yig,xi, an integer M, a small number σ (e.g. σ=0.001)1: Initialize:{β0,β,γ0,γ}={β0,β,γ0,γ}02: form=1,…,Mdo3: Update {pg(o)}m=argmax log L(θ) as in (7) given {β0,β,γ0,γ}={β0,β,γ0,γ}m−1.4: Update {β0,β,γ0,γ}m=argmax log L(θ) as in (7) given {pg(o)}={pg(o)}m. Let Lm denote the value of (7) at this step.5: if|Lm−Lm−1|≤σthen6: returnθ^={pg(o),β0,β,γ0,γ}m7: end if8: end for9: returnθ^={pg(o),β0,β,γ0,γ}MSpecifically, in the mth iteration, we obtain {pg(o)}m by maximizing the objective function (7) given {β0,β,γ0,γ}m−1. Since the integral in Equation (7) does not have a closed form, we used the Gauss–Legendre quadrature (Golub and Welsch 1969) to approximate it. Solving for {pg(o)}m is a constrained optimization problem because pg(o)∈[0,1] and ∑g=1Gpg(o)=1. To address this problem, we consider the parameterization (10)pg(o)=eug∑k=1Geuk,where ug∈R for g=1,…,G. With this reparametrization, we first obtain {ug}m using the Nelder–Mead optimization algorithm from R package optimx and then calculate {pg(o)}m from {ug}m using Equation (10). Finally, given {pg(o)}m, we obtain the optimal set {β0,β,γ0,γ}m using the Nelder–Mead optimization algorithm by maximizing log-likelihood function (7). We repeat this iterative procedure until we achieve convergence or reach the maximum number of iterations. Estimating {pg(o),β0,β,γ0,γ} for Micro-DeMix Require:yig,xi, an integer M, a small number σ (e.g. σ=0.001) 1: Initialize:{β0,β,γ0,γ}={β0,β,γ0,γ}0 2: form=1,…,Mdo 3: Update {pg(o)}m=argmax log L(θ) as in (7) given {β0,β,γ0,γ}={β0,β,γ0,γ}m−1. 4: Update {β0,β,γ0,γ}m=argmax log L(θ) as in (7) given {pg(o)}={pg(o)}m. Let Lm denote the value of (7) at this step. 5: if|Lm−Lm−1|≤σthen 6: returnθ^={pg(o),β0,β,γ0,γ}m 7: end if 8: end for 9: returnθ^={pg(o),β0,β,γ0,γ}M 2.2.2 Hypothesis testing The proposed Micro-DeMix framework also facilitates the investigation of the biodiversity of the fecal microbiome by detecting microbial groups that are differentially abundant in rectum and other GI locations. Specifically, we consider a hypothesis testing problem with H0:p(o)=p(r), where p(o) and p(r) are introduced in Section 2.1, representing the true proportions of the microbes of interest in the rectum and other GI locations, respectively. Under H0, we have ||p(o)−p(r)||2=0, motivating the following test statistic: (11)T^=(p^(o)−p^(r))⊺(p^(o)−p^(r)),where p^(o) are maximum likelihood estimators obtained from Algorithm 1 in Section 2.2, and p^(r) is given in Equation (9). Under H0:pg(r)=pg(o), we can rewrite Equation (11) as (12)T^=∑g=1G(p^g(r)−pg(r))2+∑g=1G(p^g(o)−pg(o))2−2∑g=1G(p^g(r)−pg(r))(p^g(o)−pg(o)),indicating that T^ is a function of p^(r)−p(r) and p^(o)−p(o) under H0. Thus, finding the null distribution of T^ reduces to finding the null distributions of p^(r)−p(r) and p^(o)−p(o). Specifically, applying the central limit theorem (CLT) based on model (8), we have (13)N(r)(p^(r)−p(r))→dNG(0,V(p(r))),where V(p(r))=(p1(r)(1−p1(r))−p1(r)p2(r)…−p1(r)pG(r)−p1(r)p2(r)p2(r)(1−p2(r))…−p2(r)pG(r)⋮⋱⋮−p1(r)pG(r)−p2(r)pG(r)…pG(r)(1−pG(r)))and N(r)=∑j=1S(r)Nj(r). Also, under H0, the joint density function in Equation (2) reduces to f(yig,πi|pg(o),a1,i,a2,i)=Ni!yi1!…yiG!∏g=1G{pg(o)}yig π1a1,i−1(1−πi)a2,i−1B(a1,i,a2,i)and the density function (2) becomes f(yig|·)=Ni!yi1!…yiG!∏g=1G{pg(o)}yig∫01π1a1,i−1(1−πi)a2,i−1B(a1,i,a2,i)dπi. As the integral of the beta density is 1, we have f(yig|pg(o),a1,i,a2,i)=Ni!yi1!…yiG!∏g=1G{pg(o)}yig. This indicates that under H0, the proposed beta-multinomial model in Section 2.1 reduces to a multinomial model. Indeed, when pg(r)=pg(o), Var(yig) in Remark 1 simplifies to Nipg(o)(1−pg(o)), which equals the variance of yig under the multinomial model. Thus, under H0, the maximum likelihood theory (MLE) derived from the mixture beta-multinomial model is equivalent to that derived from the multinomial model. Thus, we can apply the CLT again and obtain (14)N(p^(o)−p(o))→dNG(0,V(p(r))),where N=∑i=1SNi is the total counts of all samples collected from stool. Assuming no overlapping between stool samples and rectum samples, we establish the asymptotic distribution of T^ in the following result.Proposition 1.Let random vector (y1,y2)follow the multivariate normal distributionN2G((00),(V(p(r))00V(p(r)))). Let random vector (y1,y2)follow the multivariate normal distributionN2G((00),(V(p(r))00V(p(r)))). Suppose p(o)=p(r)andN(r)/N→K>0. ThenNT^→dg(y1,y2)asN→∞, whereg(y1,y2)=y1⊺y1+(1/K)y2⊺y2−(2/K)y1⊺y2. As a special case of Proposition 1, when K=1 or N(r)=N, we have NT^→dg(y1,y2)=y1⊺y1+y2⊺y2−2y1⊺y2. Based on Proposition 1, we develop a simulation-based algorithm for calculating the p-value, as detailed in Algorithm 2. Algorithm 2.Simulation method of testing H0:p(o)=p(r) Require: yjg(r),yig,xi , a large integer B (e.g. B=10 000)1: Estimate p^(o) and p^(r) as in Section 2.2.2: Compute T^ as in (11) using p^(o) and p^(r).3: forb=1,…,Bdo4: Simulate vector v1,v2 independently from NG(0,V(p^(r))) in (13) and (14).5: Approximate p^(r)−p(r) and p^(o)−p(o) with v1/N(r) and v2/N, respectively. Compute T^b as in (12).6: end for7: Calculate the p-value: p^←11+B(1+∑b=1B1{T^b≥T^}).8: returnp^ Simulation method of testing H0:p(o)=p(r) Require: yjg(r),yig,xi , a large integer B (e.g. B=10 000) 1: Estimate p^(o) and p^(r) as in Section 2.2. 2: Compute T^ as in (11) using p^(o) and p^(r). 3: forb=1,…,Bdo 4: Simulate vector v1,v2 independently from NG(0,V(p^(r))) in (13) and (14). 5: Approximate p^(r)−p(r) and p^(o)−p(o) with v1/N(r) and v2/N, respectively. Compute T^b as in (12). 6: end for 7: Calculate the p-value: p^←11+B(1+∑b=1B1{T^b≥T^}). 8: returnp^ 2.3 Micro-DeMix-2: an EM approach The proposed Micro-DeMix-1 procedure has strengths in easy and efficient computation: its estimation component can be implemented using standard optimization packages, and its hypothesis-testing component runs quickly due to the use of an asymptotic null distribution. However, it may have a limited scope of applicability due to the numerical evaluation of the log-likelihood in Equation (7). Specifically, based on Equation (2), we know that when {πipg(r)+(1−πi)pg(o)}yig is close to 0, which can occur for rare microbes with extremely small proportions or more common microbes with very large counts, Equation (7) may be evaluated as negative infinity. This happens because, although {πipg(r)+(1−πi)pg(o)}yig is technically not zero, it falls below the smallest representable number in standard software. To address this issue, instead of working with the marginal density function of yig in Equation (7), we develop a new estimation procedure based on the joint density function of yig and πi, as shown in Equation (2). Specifically, the joint log-likelihood is given by (15)ljoint(θ;Y,π)=∑i=1S log f(yi,πi|p(r),p(o),a1,i,a2,i)=∑i=1S∑g=1Gyig log(πipg(r)+(1−πi)pg(o))+∑i=1S(a1,i−1) log(πi)+∑i=1S(a2,i−1) log(1−πi)−∑i=1S log (B(a1,i,a2,i))+C,where C is some constant, Y=(y1,…,yS)⊺, and π=(π1,…,πS)⊺. Thanks to the logarithm, Equation (15) is always well defined, regardless of how small {πipg(r)+(1−πi)pg(o)}yig becomes. However, we cannot directly maximize Equation (15) because the πi values are not observed. To address this, we develop an EM algorithm (Dempster et al. 1977) for estimating θ based on Equation (15). We begin with an initial value θ0. In the kth E-step, we compute Qk(θ)=E[ljoint(θ,Y,π)|θk−1,yi:i=1,…,S], which is the conditional expectation of ljoint given yi:i=1,…,S and the values of θ from the previous (k−1)th step for k≥1. Unfortunately, Qk(θ) does not have a closed form. Thus, we approximate Qk(θ) using Q˜k(θ)=M−1∑m=1Mljoint(θ,Y,π˜m(k)), where π˜m(k) is a random sample from the conditional distribution of π given Y and θk−1, generated using the Metropolis–Hastings (MH) algorithm (Hastings 1970). In the kth M-step, we then obtain θk by maximizing Q˜k(θ) with existing optimization tools such as the Nelder–Mead algorithm. The final estimate θ^=(p^(r),p^(o),β^0,β^,γ^0,γ^)⊺ is obtained by iterating the EM algorithm between the E-step and M-step until convergence.Algorithm 3.Estimating {pg(o),β0,β,γ0,γ}: an EM approachRequire:yig,xi, an integer K, an integer M, a small number σ (e.g. σ=0.0001)1: Initialize:{pg(o),β0,β,γ0,γ}={pg(o),β0,β,γ0,γ}02: fork=1,…,Kdo3: (E-step) Generate a random sample π˜m(k) from the conditional distribution of π given Y and θ(k−1) using the Metropolis–Hastings (MH) algorithm. Approximate the conditional expectation Qk(θ)=E[ljoint(θ,Y,π)|θ(k−1),yi:i=1,…,S] using Q˜k(θ)=M−1∑m=1Mljoint(θ,Y,π˜m(k)),4: (M-step) Obtain θ(k) by maximizing the objective function Q˜k(θ).5: if∑‖θ(k)−θ(k−1)‖≤σthen6: returnθ^={pg(o),β0,β,γ0,γ}k7: end if8: end for9: returnθ^={pg(o),β0,β,γ0,γ}KDue to the additional variation introduced by the EM algorithm, Algorithm 2 is no longer suitable for generating p-values for testing H0:p(o)=p(r) based on the EM estimates. We fix this issue by developing a permutation-based procedure that works seamlessly with the EM algorithm; see Algorithm 4. Recall that yjg(r) denotes the absolute abundance for taxon g in the jth subject of the rectum dataset and xj(r) is the corresponding covariate. Letting yj(r)=(yj1(r),…,yjG(r))⊺, we first randomly select S individuals from {y1(r),…,ySr(r),y1,…,yS} to form a permuted stool microbiome dataset, with the remaining Sr subjects forming a permuted rectum microbiome dataset. We also obtain the permuted covariates corresponding to the permuted stool sample. Next, we use the permuted rectum sample to estimate p(r) and apply the EM algorithm to the permuted stool microbiome data and covariates to obtain an estimate of θ, denoted by θ^perm=(p^perm(r),p^perm(o),β^0,perm,β^perm,γ^0,perm,γ^perm)⊺. We then calculate the permuted test statistic Tperm=||p^perm(r)−p^perm(o)||2. This permutation procedure is repeated B times to generate B permuted test statistics Tperm(1),…,Tperm(B). Finally, the permutation p-value for testing H0:p(r)=p(o) is defined as the proportion of Tperm(1),…,Tperm(B) that are larger than or equal to the unpermuted test statistic T=||p^(r)−p^(o)||2.Algorithm 4.Permutation method of testing H0:p(o)=p(r)Require:yjg(r),yig,xi,xj(r), a large integer B1: Estimate p^(r) and p^(o) using Algorithm 3 in Section 2.3.2: Compute T^ as in (11) using p^(o) and p^(r).3: forb=1,…,Bdo4: Randomly permute data {(yj(r),xj(r))}j=1S(r),{(yi,xi)}i=1S to generate permuted rectum data and permuted stool data {(y˜j(r),x˜j(r))}j=1S(r),{(y˜i,x˜i)}i=1S.5: Estimate p^perm(r) and p^perm(o) using permutated data {(y˜j(r),x˜j(r))}j=1S(r),{(y˜i,x˜i)}i=1S.6: Compute T^perm(b)=||p^perm(r)−p^perm(o)||2.7: end for8: Calculate the p-value: p^←11+B(1+∑b=1B1{T^perm(b)≥T^}).9: returnp^Compared to Micro-DeMix-1, the proposed EM algorithm overcomes the numerical issues caused by rare microbes or large counts, giving it broader applicability. However, the EM algorithm is more computationally intensive due to its relatively slow convergence rate and the repeated use of the MH algorithm in the E-step. Additionally, the permutation test requires running the EM algorithm multiple times, making it less efficient than Algorithm 2, which relies on the asymptotic null distribution. Furthermore, as shown in the simulation studies below, Micro-DeMix-1 may have higher power than the EM algorithm in numerical applications. Estimating {pg(o),β0,β,γ0,γ}: an EM approach Require:yig,xi, an integer K, an integer M, a small number σ (e.g. σ=0.0001) 1: Initialize:{pg(o),β0,β,γ0,γ}={pg(o),β0,β,γ0,γ}0 2: fork=1,…,Kdo 3: (E-step) Generate a random sample π˜m(k) from the conditional distribution of π given Y and θ(k−1) using the Metropolis–Hastings (MH) algorithm. Approximate the conditional expectation Qk(θ)=E[ljoint(θ,Y,π)|θ(k−1),yi:i=1,…,S] using Q˜k(θ)=M−1∑m=1Mljoint(θ,Y,π˜m(k)), 4: (M-step) Obtain θ(k) by maximizing the objective function Q˜k(θ). 5: if∑‖θ(k)−θ(k−1)‖≤σthen 6: returnθ^={pg(o),β0,β,γ0,γ}k 7: end if 8: end for 9: returnθ^={pg(o),β0,β,γ0,γ}K Permutation method of testing H0:p(o)=p(r) Require:yjg(r),yig,xi,xj(r), a large integer B 1: Estimate p^(r) and p^(o) using Algorithm 3 in Section 2.3. 2: Compute T^ as in (11) using p^(o) and p^(r). 3: forb=1,…,Bdo 4: Randomly permute data {(yj(r),xj(r))}j=1S(r),{(yi,xi)}i=1S to generate permuted rectum data and permuted stool data {(y˜j(r),x˜j(r))}j=1S(r),{(y˜i,x˜i)}i=1S. 5: Estimate p^perm(r) and p^perm(o) using permutated data {(y˜j(r),x˜j(r))}j=1S(r),{(y˜i,x˜i)}i=1S. 6: Compute T^perm(b)=||p^perm(r)−p^perm(o)||2. 7: end for 8: Calculate the p-value: p^←11+B(1+∑b=1B1{T^perm(b)≥T^}). 9: returnp^ 3 Results We first investigate the finite-sample performance of Micro-DeMix using simulation. We studied the accuracy in the estimation of p(o) and the type-I error rate and power when testing for differential abundance between the gut microbiome in the rectum and other GI locations. For rectum samples, we simulated the microbial counts yj(r) from a Multinomial(Nj(r),p(r)) distribution with pg(r)=1/G for g=1,…,G and j=1,…,S(r). We next generated the stool microbiome data using the proposed beta-multinomial model. Specifically, we first set (β0,β,γ0,γ)=(0.1,0.1,0.2,0.1) and generated the covariate xi from the standard normal distribution. We then randomly generated πi from Beta(a1,i,a2,i), where a1,i and a2,i are defined in Equations (3)–(6). We assigned ug at evenly spaced values from −1 to 1 for g=1,…,G−1 and let uG=−∑g=1G−1ug. We computed pg(o) from ug according to Equation (10). Finally, we simulated the stool microbiome data yi from a Multinomial(Ni,pi) distribution where pi is defined in Equation (1) for i=1,…,S. 3.1 Simulation: Estimation We assessed the performance of Micro-DeMix in estimating p(o) under six settings, where we considered S=S(r)=100, Nj(r)=Ni=500,1000,5000 for all i and j, and G=5,10. Given the small scale of this simulation and the efficiency of the Micro-DeMix-1 algorithm, we only implemented the Micro-DeMix-1 algorithm in this section. We reported relative squared errors (RSE) to quantify the discrepancy between our Micro-DeMix estimators p^(o) and the true parameters according to RSE=∑g(pg(o)−p^g(o))2∑g(pg(o))2. The mean and standard deviation of RSE under all six settings are presented in Table 1. It was observed that as the library size increases, both the mean and standard deviation of the RSE tend to decrease. This indicates that the accuracy of estimation improves as we increase the library size. Furthermore, it is noticeable that the RSEs for G=10 are greater than those for G=5. This phenomenon occurs because the true proportions become smaller as the total number of taxa increases. Specifically, by design, pg(o)={0.059,0.116,0.225,0.439,0.161} when G=5, and pg(o)={0.031,0.039,0.051,0.065,0.083,0.107,0.137,0.177,0.227,0.083} when G=10. This makes estimation more challenging and contributes to a decrease in relative accuracy. Table 1.Mean and standard deviation of RSE obtained under each setting while estimating p(o) in Micro-DeMix. G=5 G=10 Library size5001000500050010005000Mean0.000490.000350.000140.005570.005010.00364Stand. Dev.0.000450.000350.000270.002610.002010.00111 Mean and standard deviation of RSE obtained under each setting while estimating p(o) in Micro-DeMix. We also examined the performance of Micro-DeMix-EM using a larger-scale simulation study with G=100, as detailed in the Supplementary Material. 3.2 Simulation: Type-I error rate and power In this subsection, we assessed the empirical type-I error rate and power of both Micro-DeMix testing procedures to detect differential abundance between rectal microbes and microbes from other GI locations. Accordingly, we increased the total number of microbes to G=10,20. The data-generating process was the same as that in the previous section except for a few modifications to facilitate the assessment of the type-I error rate and power. Specifically, we set (β0,β,γ0,γ)=(0.1,0.1,0.7,0.2) and generated the covariate xi from a normal distribution N(10,0.1). For each G, we generated p(r) by allowing each pg(r) and pg(o) to differ by δ: pg(r)=pg(o)+δ for g=1,…,G/2, and pg(r)=pg(o)−δ for g=G/2+1,…,G. This process ensures that ∑g=1Gpg(r)=1 for any δ. We considered relatively weak signals, i.e. δ=0,0.0004,0.0008,…,0.002, for G=10, and further decreased the signal strength by considering δ=0,0.0002,0.0004,…,0.001 for G=20. To ensure a large enough sample size and library size for signal detection, we considered S=S(r)=100, Nj(r)=Ni=30 000 for G=10 and Nj(r)=Ni=35 000 for G=20. We compared the performance of our method with seven existing differential abundance tools for microbiome data, including ANCOM-BC (Mandal et al. 2015), DESeq2 (Love et al. 2014), edgeR (Robinson et al. 2010), limma voom (Ritchie et al. 2015), MaAsLin2 (Mallick et al. 2020), t-test, and Wilcoxon test, using simulated datasets with two sample groups. The Wilcoxon test and t-test were conducted using the R functions wilcox.test and t.test, respectively. ANCOM-BC, which analyzes microbiome compositions, was performed using the ancombc function from the R package ANCOMBC. EdgeR, DESeq2, limma voom, and MaAsLin2 were implemented based on microbial counts using the R packages edgeR, DESeq2, limma, and Maaslin2, respectively. However, all these existing methods perform separate univariate tests for individual taxa, while our proposed Micro-DeMix focuses on global testing for a group of microbes. To facilitate a meaningful comparison, we applied Bonferroni correction to the univariate p-values generated by each existing method to control the family-level type-I error rate, rejecting the global null hypothesis if at least one adjusted p-value was below the significance level of α=0.05. We implemented the proposed Micro-DeMix testing procedure (detailed in Algorithm 2) to produce a p-value for testing H0:p(r)=p(o) based on 10 000 simulated datasets from the null distribution; i.e. B=10,000 in Algorithm 2. Meanwhile, we also applied the Micro-DeMix-EM testing procedure as outlined in Algorithm 4 to obtain a p-value with B=100. A smaller value of B was chosen for the permutation procedure to reduce the computational burden of Algorithm 4. Based on 100 independent replications and the significance level α=0.05, for δ=0 and δ>0, the rejection rate in Fig. 1 represents the type-I error rate and the power, respectively. The value t on x-axis represents the total signal, and some calculations yield that t=||p(r)−p(o)||22=Gδ2. The points at 0 on x-axis representing the scenario that p(r)=p(o). Type-I error rates and powers obtained at various signal strengths from the simulation study. (A) Power curves for G = 10. (B) Power curves for G = 20. A horizontal dashed line is shown at 0.05; t on the x-axis denotes the total signal. Rejection rates were obtained using both Micro-DeMix procedures and seven existing tools. As shown in Fig. 1, the power increases as the total signal grows. Micro-DeMix-EM exhibits lower power than Micro-DeMix, likely due to the slow convergence rate of the EM algorithm and/or the reduced number of permutations. Nonetheless, both Micro-DeMix procedures outperform all existing methods, except for ANCOM-BC in terms of power, but ANCOM-BC exhibits the highest type-I error rate. This result highlights that Micro-DeMix offers the best balance between type-I error rate and power among the methods evaluated. The reason most existing methods lack power is that they overlook the fact that the stool microbiome is a mixed population of rectum microbes and microbes from other GI locations. ANCOM-BC’s distinct performance warrants further benchmarking studies to better understand its unique behavior. We next revisit the iHMP IBD cohort, which is briefly discussed in Section 1, to demonstrate the effectiveness and efficiency of the proposed Micro-DeMix in elucidating the composition of real fecal microbiome data. We are particularly interested in understanding the heterogeneity of the stool microbiome, how the stool microbiome differs from the rectum microbiome in IBD populations, and how that difference may be related to the pathology of IBD. As discussed in Section 1, the IBD cohort was a longitudinal study, collecting taxonomic data on the microbiome from 132 individuals (104 IBD patients) by analyzing 1785 stool samples and 651 intestinal biopsies over the course of 1 year (Integrative HMP et al. 2019). This IBD cohort was part of the iHMP, which was designed to explore host–microbiome interplay to gain a holistic view of host–microbe interactions (Integrative HMP et al. 2019). The IBD IBDMDB research team collected microbiome data from different sites within the human gut of individuals with IBD to understand how the microbiome impacts human health and disease. For a fair comparison, our analysis focused on stool and rectum samples collected at the baseline, resulting in 13 samples for stool and 38 samples for rectum data in IBD patients. After excluding subjects with missing covariates of interest, such as age and gender, the stool group was narrowed down to 12 subjects. The operational taxonomic unit (OTU)-level data for both groups are highly zero-inflated. More than 90% of the entries in the OTU table derived from the stool samples are zero. Therefore, in the results below, we took a hierarchical approach to investigate the microbial composition of the fecal microbiome as compared to the rectum microbiome at higher taxonomic levels to mitigate the potential influences of excess zeros. We also noticed significant variations in total reads (library size) across samples for both groups, specifically, ranging from 1499 to 79 228 for stool samples and from 34 to 31 781 for rectum samples. This nonbiological variation can result in samples with disproportionately high read counts dominating the analysis under the multinomial model, potentially masking true biological patterns due to this technical factor. We mitigated this issue by performing rarefaction, using the R function rarefy_even_depth from package phyloseq as in the exiting literature (McMurdie and Holmes 2013). In microbial studies, such rarefying has been used for normalization by randomly subsampling the data to mitigate the potential influence of varying library sizes (Hughes and Hellmann 2005, Koren et al. 2013, Navas-Molina et al. 2013). We also acknowledge that there is an ongoing debate in the research community regarding the use of rarefaction; see McMurdie and Holmes (2014) for more details. Given this, we suggest that the decision to perform rarefaction in microbiome data analysis should be carefully considered based on the data characteristics, research goals, and biological interpretations. We also applied the Micro-DeMix-EM algorithm to the IBD data without performing rarefaction, and the results are presented in the Supplementary Material. 3.3 iHMP IBD data: Phylum-level analysis We aggregated the absolute counts from the OTU level to the phylum level, resulting in eight phyla, leading to a much lower percentage of zeros compared to the OTU-level data (50.9% versus 93.6%). We used 1499 as the rarefaction level for stool samples, as it is the minimum library size in the stool group. For a fair comparison, we applied the same rarefaction level to rectum samples, resulting in 30 samples in rectum group. Since this procedure is based on random subsampling without replacement, we repeated the procedure 100 times and took the average to reduce variability for both groups. Samples with library sizes less than 1499 were removed from the analysis. We computed the relative abundance of these eight phyla in both the rectum and stool samples, and the results revealed a clear difference in the microbiome composition (Fig. 2). Relative abundance of microbial populations at the phylum level in the rectum, stool, and other GI locations. Phylum Firmicutes was highly abundant in both the rectum and stool. It was the major phylum detected, representing more than 50% of the total microbial relative abundance in both groups. Overall, the relative abundance of Firmicutes in the stool (∼61.4%) was slightly higher than that in the rectum (∼52.0%). When compared to the microbial profile in rectum, the stool samples exhibited a considerable increase in the relative abundance of phylum Proteobacteria (from 7.1% to 30.1%), while the relative abundance of Bacteroidetes decreased from 38.1% to 6.7%. In addition, we observed an elevated relative abundance of the Verrucomicrobia phylum in the stool microbiome compared to the rectum, with an increase from 0.31% to 1.5%. Conversely, the phylum Fusobacteria showed a decrease from over 1.9% to a lower level. The remaining three phyla were nearly imperceptible in rectum and stool. To further understand the composition of the fecal microbiome in IBD populations, we fitted the proposed Micro-DeMix model, incorporating gender and age as covariates. This analysis revealed estimates of the relative abundance of selected phyla from the other GI locations (Fig. 2). Compared to the rectum microbiome, the microbial profile of other GI locations revealed a higher relative abundance of phylum Firmicutes (∼64.3%), Proteobacteria (∼33.7%), and Verrucomicrobia (∼1.7%). Phylum Bacteroidetes was undetectable in other GI locations. The remaining phyla also exhibited relatively low abundance (<1%). Based on 10 000 simulated datasets, the Micro-DeMix test suggested significant differences in the taxonomic composition of the eight phyla between the rectum and other GI locations in IBD populations (p-value <1e−4). Our findings are consistent with recent studies on the gut microbiome in IBD populations. Specifically, the rectum and fecal microbiome of IBD patients is dominated by three major bacterial phyla: Firmicutes, Bacteroidetes, Proteobacteria, and to a lesser degree Verrucomicrobia and Actinobacteria (Frank et al. 2007, Lo Presti et al. 2019). Our results are also consistent with recent studies that described distinct changes in the composition of the microbiota along the GI tract in IBD populations. The lower GI tract exhibited a larger abundance of Firmicutes and Bacteroidetes, whereas the upper GI tract was predominated by Proteobacteria and Firmicutes (Vuik et al. 2019). Compared to healthy individuals, the reduction in the relative abundance of Firmicutes and an elevation in Bacteroidetes have been observed in IBD patients (Khan et al. 2019, Lo Presti et al. 2019, Vuik et al. 2019, Zhang et al. 2022). Thus, our results shed light on the scientific premise of developing an effective microbiome-based treatment approach for IBD, such as the therapeutic supplementation of probiotics, prebiotics, and symbiotics, and fecal microbiota transplantation. 3.4 iHMP IBD data: Microbial composition within Proteobacteria and Firmicutes In Section 4.1, we identified Proteobacteria and Firmicutes as the most abundant phyla in stool samples. To further understand the microbes in these two phyla, we conducted analyses of microbial composition at lower taxonomic levels (class, order, family) within Proteobacteria and Firmicutes phylum (Fig. 3). Relative abundance of microbial populations at various taxonomic levels in the rectum, stool, and other GI locations. (A, B) Microbial composition at the class and order levels within the phylum Proteobacteria. (C, D) Microbial composition at the order and family levels within the phylum Firmicutes. Similar to the analyses in Section 4.1, we computed the relative abundance of all microbial classes and orders within phylum Proteobacteria for both the stool and rectum samples, and implemented the Micro-DeMix model to elucidate the difference between the rectum microbiome and the microbes in other GI locations. We followed the rarefying procedure that described in Section 4.1 to process the stool OTU data. In stool samples, the minimum library size is 13, which is not big enough to be included in the study. Therefore, we used 369, the second minimum, as the rarefaction level for both stool and rectum group. The outcomes of our analyses are visually represented in Fig. 3A and B. Fig. 3A reveals that within phylum Proteobacteria, the major classes in rectum and stool are Betaproteobacteria and Gammaproteobacteria. However, the rectum microbiome is distinguished by a higher relative abundance of class Deltaproteobacteria (∼11.1%) and Alphaproteobacteria (∼3.0%), which are significantly low in abundance in stool samples. There is a low relative abundance of Epsilonproteobacteria (<1%) in both stool and rectum. As seen in Fig 3A, Gammaproteobacteria (∼99.6%) is the most dominant class among the selected classes in other GI locations, while the other four classes are nearly undetectable. Although this finding may appear extreme, it is aligned with the existing knowledge that IBD patients have an altered gut microbiota marked by an increased relative abundance of Gammaproteobacteria class (Scales et al. 2016). We proceeded to analyze the microbial composition of the phylum Proteobacteria at the order level, revealing marked differences between the rectum and stool samples (Fig. 3B). While both showed high abundance in orders Burkholderiales, Enterobacteriales, and Pasteurellales, the rectum samples were mainly characterized by the orders Desulfovibrionales and Neisseriales. We utilized the Micro-DeMix method to assess the microbiome composition in other GI locations. Our analysis revealed a dominance of Enterobacteriales (∼81.1%), along with a lower proportion of Pasteurellales (∼18.83%). This finding is supported by a recent study showing that Enterobacteriaceae exhibit increased abundance in the low-pH GI location and ileum (Morgan et al. 2012). The Micro-DeMix test indicates significant differences in microbial profiles between the rectum and other GI locations (p-value <1e−4) within the phylum Proteobacteria at both the class and order levels. We further explored the microbial composition of the Firmicutes phylum. In the stool samples, three classes were identified: Bacilli, Clostridia, and Erysipelotrichi. At the original OTU level, class Clostridia dominated with a percentage of 97.1%, while class Erysipelotrichi was nearly undetectable. Given these findings, we narrowed our focus to lower taxonomic levels, specifically, order and family within the Firmicutes phylum. Similarly, we employed the minimum library size, 493, as the rarefaction level and processed both stool and rectum OTU data. The corresponding results are shown in Fig. 3C and D. Notably, at the order level (Fig. 3C), the phylum Firmicutes in the rectum and stool was predominantly represented by the order Clostridiales (97.1% in rectum and 93.6% in stool of all Firmicutes). Our results demonstrate the presence of order Turicibacterales in the stool samples, while it is absent in the rectum samples, which suggest that order Turicibacterales in stool samples may originate from other GI locations. The relative abundance of the order Lactobacillales increased in stool compared to rectum from 0.86% to 5.7%. In contrast, order Erysipelotrichales is found to be present (2.0%) in the rectum but remains at lower levels in the stool. The results of Micro-DeMix analysis show a p-value <1e−4 for testing differential abundance between the rectum and other GI locations. Meanwhile, it highlights a significantly higher relative abundance of Lactobacillales (∼63.4%) in other GI locations compared to the rectum and stool. The analysis estimates that order Turicibacterales (∼6.4%) is also more abundant in other GI locations. These findings indicate a distinct microbial composition in other GI locations, with increased prevalence of orders Lactobacillales and Turicibacterales. At the family level within the Firmicutes phylum (Fig. 3D), major families of the rectum microbiota are Ruminococcaceae, Lachnospiraceae, and Veillonellaceae. Comparisons between stool and rectum samples reveal an increase in Ruminococcaceae and a decrease in Lachnospiraceae and Veillonellaceae, aligning with findings from other studies (Lo Presti et al. 2019). One significant finding is the presence of the Clostridiaceae, Enterococcaceae, Planococcaceae, and Turicibacteraceae families in stool samples but not in the rectum. This observation suggests that these families detected in stool originate from other GI microbiome community. In other GI locations, our analysis shows a higher abundance of the Clostridiaceae and Ruminococcaceae families. In contrast, the family Lachnospiraceae exhibits a comparatively lower abundance. Furthermore, the p-value obtained through Micro-DeMix hypothesis testing is low (<1e−4). These findings imply distinctive microbial profiles among the rectum, stool, and other GI locations, indicating variations in the relative abundance of specific taxa. 4 Conclusion In this paper, we introduced Micro-DeMix, a beta-multinomial model designed for deconvoluting microbial abundance in stool samples. We proposed two parameter estimation procedures for the model, each with its strengths in specific types of applications. Additionally, we developed hypothesis testing procedures for detecting differential abundance, which has proven more effective compared to existing methods. We applied Micro-DeMix to rectum and stool microbiome datasets from the iHMP IBD cohort in elucidating the composition of real fecal microbiome data in IBD populations. In the analyses of the iHMP IBD data, we performed rarefying on stool and rectum datasets to address the large variations in library sizes across samples. While rarefaction has been widely adopted in real microbiome applications, a limitation of this approach is that the selection of rarefaction level is data-dependent, and different rarefaction levels may lead to slightly different results. Due to the limited sample size, our strategy for selecting the rarefaction level is to keep as many samples as possible while removing samples with unreliably low library sizes. Different strategies may be considered for other datasets. As mentioned earlier, the current paper focuses on integrating rectum samples and fecal samples due to data availability and biological relevance. However, the proposed method can seamlessly be used for integration of fecal samples with microbiome data collected from other GI locations. Furthermore, with microbiome data available from two or more GI locations, we could extend Micro-DeMix to have a refined decomposition of the fecal microbiome. Specifically, following work could consider the stool microbiome as a J-component mixture of gut microbes, i.e. we let pig=πi(1)pg(1)+⋯+πi(J−1)pg(J−1)+(1−πi(1)−⋯−πi(J−1))pg(J) in Equation (1), where πi follows a Dirichlet (ai) distribution. We leave this fruitful area for future research. Supplementary Material