Two-Stage Multiple Test Procedures Controlling False Discovery Rate with auxiliary variable and their Application to Set4 Δ \Delta Mutant Data Seohwa Hwang, Mark Louie Ramos, DoHwan Park, Junyong Park, Johan Lim, Erin Green Abstract In this paper, we present novel methodologies that incorporate auxiliary variables for multiple hypotheses testing related to the main point of interest while effectively controlling the false discovery rate. When dealing with multiple tests concerning the primary variable of interest, researchers can use auxiliary variables to set preconditions for the significance of primary variables, thereby enhancing test efficacy. Depending on the auxiliary variable’s role, we propose two approaches: one terminates testing of the primary variable if it does not meet predefined conditions, and the other adjusts the evaluation criteria based on the auxiliary variable. Employing the copula method, we elucidate the dependence between the auxiliary and primary variables by deriving their joint distribution from individual marginal distributions.Our numerical studies, compared with existing methods, demonstrate that the proposed methodologies effectively control the FDR and yield greater statistical power than previous approaches solely based on the primary variable. As an illustrative example, we apply our methods to the Set4 Δ \Delta mutant dataset. Our findings highlight the distinctions between our methodologies and traditional approaches, emphasising the potential advantages of our methods in introducing the auxiliary variable for selecting more genes. \DOIsuffix bimj.200100000 \Volume 52 \Issue 61 \Year 2024 \pagespan 1 \Receiveddate zzz \Reviseddate zzz \Accepteddate zzz keywords Copula; False Discovery Rate; Multiple Testing; Two-Stage Procedure; 1 Introduction In this paper, we consider multiple testing scenarios where additional information beyond p-values is available. Numerous studies have already tackled this subject, predominantly focusing on enhancing rejection rates through the utilisation of covariates. For instance, 10 demonstrated that regardless of covariate values, p -values under the null hypothesis conform to a uniform distribution. In contrast, AdaFDR, proposed by 17 , relaxes the requirement of a uniform distribution under the null hypothesis but mandates a mirroring property in the distribution of p -values given specific covariates, which means the conditional null distribution of p -value given the covariate is symmetric around 1/2 . 8 explore maximising rejections by assigning varying weights to groups based on covariates, while 1 estimate the proportion of alternative hypotheses given covariates to adjust the rejection region. On the other hand, 2 proposed a multiple testing procedure when there is auxiliary information such as the order of hypotheses or alternative distribution while there is no specific covariate. These methodologies aim to optimise rejection rates while ensuring control over the False Discovery Rate (FDR) across all hypothesis tests. While maximising the number of rejections with controlling an error rate is statistically advantageous, there are instances where such strategies may not be desirable when considering scientific knowledge or background. We incorporate the auxiliary variable more relevantly through modelling the joint distribution of the p -value and the auxiliary variable so that we do not require such strong assumptions such as uniform distribution of p -value under the null hypothesis regardless of the covariate or mirroring property of p -value under the null hypothesis. One main aim is to incorporate an auxiliary variable to improve the results of multiple testing compared to the case of using only the primary variable. In addition, scientists may reflect some prior knowledge on the main interest through the auxiliary variable. Some restrictions on the distribution of p value given the value of the covariate may not reflect such flexible situations. The reflection of the auxiliary variable in the test of the primary variable essentially signifies that there is a dependency between the two variables, and identifying this dependency is crucial. We emphasise that only the primary variables are tested and the auxiliary variables provide supplementary information to make better decisions on the primary variables. In this sense, our problem is different from the case of bivariate p -values which are declared to be significant when at least one of p -values is significant. See 4 . Rather than imposing the restrictions on the distribution of the p value, we use the joint distribution of the auxiliary and primary variables and then propose two types of two-stage procedures controlling FDR. One is called two-stage FDR(H) based on the idea of hard thresholding such as we filter insignificant genes using the auxiliary variable and then only if this stage is passed, the significance of the primary variable is considered. The other one is two-stage FDR(S) where (S) presents the soft threshold which avoids filtering. Two-stage FDR(S) is testing of the primary variable with adjusted rejection region for the primary variable reflecting the result for the auxiliary variable. As mentioned, our goal is to test only the primary variable while reflecting the information coming from the dependence between the primary and auxiliary variables. The proposed methods are demonstrated through simulation to effectively control the FDR while achieving superior testing power compared to existing methods. Existing methods incorporating the auxiliary variable with the restrictions on the distribution of p value have difficulty in controlling a given level of FDR and obtaining powers due to the constraints such as the uniform distribution of p value regardless of the value of covariate and the mirroring property under the null hypothesis. In the context of Set4 \Delta dataset, the two-stage FDR(S) approach successfully identifies a cluster of genes that not only possess statistical significance but are also known as significant from biological experiments. This paper is organised as follows. In section 2, we present the description of Set4 \Delta mutant data, which serves as the motivation for this study, along with the necessity of the two-step method proposed. Section 3 introduces the copula that explains distribution estimation and dependence, which are the basic items required for the proposed methods, and explains the joint distribution estimation procedure. Section 4 describes two methodologies based on the two-step procedures designed for controlling FDR. Section 5 provides simulation studies to compare the existing methods with the proposed two-stage procedures and section 6 presents an analysis of Set4 \Delta mutant data. Finally, Section 7 contains concluding remarks. 2 Motivating Data set: Set4 \Delta mutant data The motivating study seeks to identify genes within the genome of the budding yeast Saccharomyces cerevisiae that are differentially expressed when a particular gene expression machinery referred to as Set4 is mutated. The data set is referred to as the Set4 \Delta mutant data set. In order to identify the genes that depend on Set4 during stress, the gene expression profile of the whole genome in cells where Set4 is silenced (knock-out condition or KO) and cells where it is not (wildtype condition or WT) are determined. In addition, cells are treated with hydrogen peroxide to induce oxidative stress. The experiment is replicated thrice. In each replication, the amount of mutation, represented by a weighted count value, was collected for each of about 8,000 RNA species under KO and WT conditions. For each gene, these weighted counts are averaged to produce a single value under each condition. More formally, we have the data for M genes and three replicates such as \displaystyle WT_{ij},\quad KO_{ij} (1) under WT and KO conditions, respectively for 1\leq i\leq M and 1\leq j\leq 3 . Let \mu_{WT_{i}} and \mu_{KO_{i}} be the mean of W_{ij} and K_{ij} , respectively, then we define the vector of logfold changes of M genes as \displaystyle\beta^{\prime}=(\beta_{1},\ldots,\beta_{M})^{\prime}=\left(\log\frac{\mu_{WT_{1}}}{\mu_{KO_{1}}},\ldots,\log\frac{\mu_{WT_{M}}}{\mu_{KO_{M}}}\right)^{\prime}. (2) From these data matrices, the vector of statistics that is to be used for multiple testing is \hat{\beta}^{\prime}=(\hat{\beta}_{1},\ldots,\hat{\beta}_{M})^{\prime}=\left(\log{\frac{\bar{WT_{1.}}}{\bar{KO_{1.}}}},\cdots,\log{\frac{\bar{WT_{M.}}}{\bar{KO_{M.}}}}\right)^{\prime} (3) where \bar{WT_{i.}}=\frac{1}{3}\sum_{j=1}^{3}W_{ij} and \bar{KO_{i.}}=\frac{1}{3}\sum_{j=1}^{3}KO_{ij} are the sample means from 3 replicates for the i th gene. The logfold changes statistic is then computed as \displaystyle\widehat{\beta_{i}}=\log_{2}\frac{\bar{KO}i}{\bar{WT}}{i} (4) where i=1,2,\cdots M and M is the total number of genes. \bar{WT} is a mean weighted count measure of mutations under wild type condition and \bar{KO} is a mean weighted count measure of mutations under knock-out condition for the i th gene. This statistic \hat{\beta}_{i} represents the doubling effect” of one condition relative to the other condition. A gene is considered interesting” if the weighted count measure is very different between the KO and WT settings, and so correspondingly, if \mid\hat{\beta}_{i}\mid\gg 0 , we have supporting evidence for significance of the gene. 12 selected genes based on \hat{\beta}_{i} s considering multiple testing procedures controlling FDR. While logfold change is our primary focus, 12 presented the results that the genes selected based solely on logfold changes are not biologically significant in most cases. Even if logfold change is considered significant for various reasons, such as extremely small values or high variance in the data across conditions, it can be influenced by various errors occurring during the experimental process. Therefore, when considering variability in experimental replicates, it is essential to first assess the significance of changes in logfold changes values. In our analysis, the standard deviation of the logfold changes will serve as an auxiliary variable. The distributions of logfold changes and their standard deviations are shown in Figure 1 . Figure 1: The histograms show that the distributions of (A) the logfold changes and (b) their estimated standard deviations. While the exact distribution of each logfold change remains unknown, it’s reasonable to assume that different sizes of standard deviation of logfold changes affect the determination of the significance of the logfold change. Based on this idea, we propose two methods for setting rejection criteria based on standard deviation. The first method adjusts rejection regions according to whether the standard deviation exceeds a certain threshold, similar to a diagnostic screening process. The second method varies the rejection regions with the exact value of the standard deviation, allowing precise adjustments when detailed data on both the primary and auxiliary variables are available. 3 Models and Estimation of Distributions The traditional multiple testing methods with existing covariate either assume a uniform distribution under the null hypothesis regardless of the covariates or satisfy the mirroring property, where the role of the covariate is to maximise the rejection of hypotheses by adjusting rejection regions for different values of the covariate. More specifically, in the existing studies such as 10 , 17 , 8 and 1 , the null density of p value given the auxiliary variable y is either I_{[0,1]}(p) regardless of y or the mirroring property such as P(p\leq t|y)=P(p\geq 1-t|y) for t\in[0,1/2] . Figure 2: Comparison of p -value distributions: (A) p -values generated symmetrically around 0.5, and (B) p -values generated from Set4 \Delta data It’s worth noting that the mirroring property fails in some cases. This property implies a zero correlation between the p value and the auxiliary variable, often indicating a weak relationship between them. Achieving this constraint can be overly restrictive if the auxiliary variable is to be used to increase the power of the test. For example, we generated exemplary p -values that are symmetric around 0.5, as shown in Figure 2 (A). This distribution of p -values is suitable for existing statistical methods that assume such symmetry in their analyses. Conversely, p -values generated from real data typically do not exhibit this symmetry around 0.5 , which implies that the underlying assumptions of existing methods are not met. Rather than those constraints on the null distribution of p value given the auxiliary variable, our proposed methods are based on considering the joint distribution of the primary variable of interest and an additional auxiliary variable. We estimate the marginal distribution for each variable and derive the joint distribution using the copula method, based on which we propose two methods. In other words, we treat the auxiliary variable as a random variable rather than covariate so that we can consider dependent structure between those two variables. Our proposed methods do not request that the marginal distribution of p value is uniformly distributed or has mirroring property. We use the relationship between the distribution of p value and the auxiliary variable through the joint distribution of them. Therefore, our proposed methods reflect those two variables more flexibly compared to existing methods. We will demonstrate that these proposed methods yield different results compared to the existing methods. Suppose we have primary and auxiliary variables denoted by \beta and y respectively. The primary variable, \beta , is modelled as a mixture of two probability density functions, with each being assumed to have been generated under null and alternative scenarios, labelled as f_{0} and f_{1} , respectively. Thus, it is of interest to test M hypotheses of the primary variables, \displaystyle H_{0i}:\beta_{i}\sim f_{0}~~vs.~~H_{1i}:\beta_{i}\sim f_{1} (5) for 1\leq i\leq M where each \beta_{i} is modelled as \displaystyle f(\beta) \displaystyle= \displaystyle p_{0}f_{0}(\beta)+(1-p_{0})f_{1}(\beta) (6) where p_{0}=P(\mbox{$H_{0i}$ is true}) . This setting has been used in a variety of multiple testing problems. For example, 5 used a normal distribution for f_{0} of which the mean and standard deviation are estimated based on the zero assumption and the marginal distribution f is estimated using Poisson regression instead of estimating the alternative density f_{1} . In 12 , the logfold changes statistics for Set4 \Delta that come from f_{0} were modelled as a mixture of two normal distributions rather than a single normal distribution to explain two different sources of the null distribution. Once we estimate f_{0} , we also obtain the cumulative distribution function (cdf), \hat{F}_{0}(\beta)=\int_{-\infty}^{\beta}\hat{f}_{0}(t)dt . The use of an auxiliary variable ( y ) is also considered. Since the auxiliary variable is used as supplementary information, we only need to identify the marginal distribution of y such as \displaystyle y\sim h(y) (7) where h(y) is the probability density function (pdf) of y . In practice, h(y) or the cumulative distribution function H(y)=\int^{y}_{-\infty}h(t)dt can be estimated by nonparametric approaches, for example, we estimate \displaystyle\hat{H}(y)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{I}(y_{i}\leq y) (8) which is the empirical cdf of y_{i} s. This method helps researchers by allowing them to use extra scientific information that isn’t the main focus of their study, but still makes their results stronger and more trustworthy. By estimating \hat{H}(y) , the empirical cumulative distribution function of y , we apply a non-parametric approach without making restrictive assumptions about its distribution. Once we have the marginal distributions of the primary and auxiliary variables, we use copula models to estimate the dependence structure of y and \beta of which the joint distribution is G_{0}(y,\beta) . The selection of an appropriate copula is a crucial step in modelling the dependence structure in multivariate distributions. There are several approaches to choose an appropriate copula. One ad-hoc approach is based on a scatterplot of the data to gain insight into the dependence structure which can be done by converting the data using the data’s marginal cumulative distribution. There are typical patterns of scatterplots for different copulas, so we may choose the closest pattern of the scatterplot. Another approach is comparing Log-Likelihood (LogLik), Akaike Information Criterion (AIC), and Bayesian Information Criterion (BIC). These criteria are widely used for the selection of copulas. For example, see 15 and 7 . We present examples of copulas in Figure and numerical studies of the performances of these criteria for selection of copula in Table in supplementary material. In this section, we present our proposed methods depending on how we handle the auxiliary variable. We first define \delta_{i}=I({H_{0i}~\mbox{is rejected}}) and \theta_{i}={I}(H_{1i}~\mbox{is true}) where I(\cdot) is an indicator function. Our proposed methods are designed to control the FDR which is \displaystyle FDR=E\left(\frac{V}{R\vee 1}\right) (9) where V=\sum_{i=1}^{M}\delta_{i}(1-\theta_{i}) is the number of false rejections and R=\sum_{i=1}^{M}\delta_{i} is the number of rejections. We introduce the marginal p values and then propose two types of methods with some theoretical justification. 3.1 Calculation of Marginal p -values We define p_{1i} and p_{2i} which are the marginal p -values corresponding to the auxiliary variable and the primary variable, respectively. For observed y_{i} and \hat{\beta}_{i} , \displaystyle p_{1i} \displaystyle= \displaystyle H(y_{i}), (10) \displaystyle p_{2i} \displaystyle= \displaystyle 2\min\left(F_{0}(\hat{\beta}_{i}),1-F_{0}(\hat{\beta}_{i})\right). (11) We mentioned how f_{0} is estimated in the previous section. Based on the estimator of f_{0} , we have an estimator of F_{0}(\beta)=\int_{-\infty}^{\beta}f_{0}(t)dt . We also have an estimator of H(y) such as the empirical cdf as discussed in the previous section. We assume that p_{1i}\sim U[0,1] and p_{2i}\sim U[0,1] under the H_{0i}:\beta_{i}\sim f_{0} . Since (y_{i},\hat{\beta}_{i}) are generated from the joint distribution G_{0} , p_{1i} and p_{2i} are dependent. Remark 1 Note that p_{2i} covers the case of two-tailed p -value for non-symmetric distribution. For one-sided cases, we can define F_{0}(\hat{\beta}_{i}) and 1-F_{0}(\hat{\beta}_{i}) for left-tailed and right-tailed cases, respectively. Similarly, p_{1i} can also be defined for two-tailed cases. Depending on the marginal p -values such as p_{1i} and p_{2i} , we propose two methods, namely two-stage FDR(H) and two-stage FDR(S) in the following section. 3.2 Two-stage FDR(H)and Two-stage FDR(S) As mentioned before, our proposed methods are based on modelling the joint distribution of p_{1i} and p_{2i} rather than assuming p_{1i} is U(0,1) regardless of the auxiliary variable p_{2i} or the mirroring property such as the density of p_{1i} is given x under the null hypothesis is symmetric around 1/2 . We present two proposed methods, called Two-Stage FDR(H) and Two-Stage FDR(S). Two-stage FDR(H): For the first type of two-stage procedure two-stage FDR(H), we proceed as follows: Given a value of \gamma_{1} , if p_{1i}>\gamma_{1} equivalently y_{i}>H^{-1}(\gamma_{1}) , we conclude that the first requirement via the auxiliary variable is not satisfied and hence we do not proceed with testing the significance of the primary variable. On the other hand, if p_{1i}\leq\gamma_{1} equivalently y_{i}\leq H^{-1}(\gamma_{1}) , then we test the significance of the primary variable. Here, \gamma_{1} is a tuning parameter that serves as a cut-off value to determine whether to proceed with testing the primary variable. We will discuss in section 3.3 how we choose \gamma_{1} to maximise the number of discoveries. Considering this two-stage procedure, we define the p -value, p_{i} aggregating two p_{1i} and p_{2i} as follows: for a given \gamma_{1}\in[0,1] , \displaystyle p_{i}^{H}(\gamma_{1})=\begin{cases}C(\gamma_{1},p_{2i})&\quad\text{if }p_{1i}\leq\gamma_{1}\\ p_{1i}&\quad\text{if }p_{1i}>\gamma_{1}\end{cases} (12) where \displaystyle C(a,b)=P_{G_{0}}(P_{1i}\leq a,P_{2i}\leq b) (13) is a 2-dimensional copula distribution evaluated at (a,b) . Here, G_{0} stands for the joint distribution of y_{i}^{\prime}s and \beta_{i}^{\prime}s under null hypothesis. In order to reject the null hypothesis for the primary variable \beta_{i} , we need two steps such as p_{1i}\leq\gamma_{1} and a small p_{2i} , i.e., the event of {p_{1i}\leq\gamma_{1},p_{2i}\leq\gamma_{2}} for some \gamma_{2} . In testing the primary variable, it is better not to use \gamma_{2}=\alpha since p_{1i}\leq\gamma_{1} affects the test for the primary variable. More specifically, take \gamma_{2} which should satisfy the uniform distribution of P_{i} such that P_{G_{0}}(P_{1i}\leq\gamma)=P_{G_{0}}(P_{1i}\leq\gamma_{1},P_{2i}\leq\gamma_{2})=\gamma . This means \gamma_{2} depends on \gamma_{1} and \gamma , say \gamma_{2}\equiv\gamma_{2}(\gamma_{1},\gamma) . Using the definition of ( 13 ), if \hat{\beta}_{i}\geq D_{0.5} , then we have \displaystyle C(\gamma_{1},\gamma_{2})=P_{G_{0}(y,\beta)}(P_{1i}\leq\gamma_{1},P_{2i}\leq\gamma_{2})=\int^{C_{1-\gamma_{1}}}_{0}\int_{D_{1-\gamma_{2}/2}}^{1}G_{0}(y,\beta)d\beta dy=\gamma (14) where C_{\alpha} and D_{\alpha} are \alpha quantiles of f_{0} and h_{0} , respectively.; if \hat{\beta}_{i}\gamma_{1} , the corresponding primary variable y_{i} as a whole is not considered significant although the second p-value, p_{2i} , is very small. In other words, such a thresholding in the first step plays a role of screening genes which do not satisfy the condition based on the auxiliary variable y_{i} . To highlight this characteristic, we will refer to it as “two-stage FDR(H)”, denoting the implementation of two-stage FDR with hard thresholding. From two-stage FDR(H), it is crucial to check whether p_{i} has a desirable property such that p_{i}^{H} has the uniform distribution under G_{0} . Before we state Theorem 3.1 showing the uniform distribution of p_{i} in both ( 12 ) and ( 18 ) under G_{0} , we present the following lemma which is used in the proof of Theorem 3.1 . Lemma 1 From the definition of p_{i}^{H}(\gamma_{1}) in ( 12 ), we have \displaystyle P(P_{1i}\leq\gamma_{1},P_{2i}\leq\gamma_{2})=P(P_{i}^{H}(\gamma_{1})\leq\gamma,P_{1i}\leq\gamma_{1}) (17) for \gamma=C(\gamma_{1},\gamma_{2}) in ( 16 ) where \gamma_{1} and \gamma_{2} are in [0,1] . See the supplementary material. ∎Using Lemma 1 , we present the following theorem showing that p_{i}^{H}(\gamma_{1}) is uniformly distributed regardless of the choice of \gamma_{1} . \theoremname 3.1 If (\hat{\beta}_{i},y_{i})\sim G_{0}(\beta,y) , p_{i}^{H}(\gamma_{1}) ’s defined in ( 12 ) follow the uniform distribution in (0,1) . See the supplementary material. ∎ Two-stage FDR(S): Instead of this stringent thresholding procedure in two-stage FDR(H), we also provide the following method, two-stage FDR(S). The basic idea is to adjust the distribution of the second p_{2i} based on the given value of the first p_{1i} , by using the estimated copula. In fact, this will be implemented by computing the conditional distribution of p_{2i} given p_{1i} . More specifically, p_{i}^{S} aggregating p_{1i} and p_{2i} is defined as follows: \displaystyle p_{i}^{S}=C(p_{2i}|p_{1i})\equiv P_{G_{0}}(P_{2i}\leq p_{2i}|P_{1i}=p_{1i}) (18) which is the conditional cumulative distribution of p_{2i} given p_{1i} . As in two-stage FDR(H), we can show that p_{i}^{S} defined in ( 18 ) also follows uniform distribution under G_{0} . \theoremname 3.2 If (\hat{\beta}_{i},y_{i})\sim G_{0}(\beta,y) , p_{i}^{S} ’s defined in ( 18 ) follow the uniform distribution in (0,1) . See the supplementary material. ∎ Comparision of Two Methods: Two-stage FDR(S) differs from two-stage FDR(H) by introducing multiple thresholds. Notably, two-stage FDR(S) does not require a tuning parameter such as \gamma_{1} in two-stage FDR(H). Instead of relying on one threshold \gamma_{1} in two-stage FDR(H), two-stage FDR(S) uses p_{1i} to adjust the decision rule for p_{2i} through the conditional probability. We call this process as soft thresholds in the sense that we adjust rejection region for the primary variable in a smooth way which is demonstrated in Figure 3 . To emphasise the contrast with “two-stage FDR(H)”, we refer to this procedure as “two-stage FDR(S)”, representing the concept of a soft margin. To clarify the differences between two-stage FDR(H) and two-stage FDR(S), we present Figure 3 including (A), (B) and (C) for two methods, respectively. In each figure, we consider three distinct points, A , B and C which are corresponding to (p_{1i},p_{2i})=(0.4,0.1) , (0.8,0.1) and (0.1,0.143) . In Figure 3 , (A) and (B) are the cases of rejection regions for different \gamma_{1} values, \gamma_{1}=0.9 and 0.7 respectively. We see that the rejection regions have rectangular forms which means that the cut-off value for p_{2i} is identical whenever p_{1i}<\gamma_{1} . From (A) and (B), the rejections for B and C are different for different \gamma_{1} values. When \gamma_{1}=0.9 , all A , B and C are proceeded to test the primary variable p_{2i} , however C is not rejected since p_{2i} is larger than the cut-off value. On the other hand, when \gamma_{1}=0.7 , B does not move to the testing of the primary variable, however C is rejected to be significant. This shows that there is trade-off relationship between \gamma_{1} and the cut-off value for p_{2i} . For larger value of \gamma_{1} (equivalently, weak requirement on the auxiliary variable), the cut-off value for p_{2i} is small, i.e., more stringent on testing the primary variable. Conversely, for small value of \gamma_{1} (stringent on the auxiliary variable), testing for the primary variable is less stringent, i.e., the cut-off value is larger for p_{2i} . Regarding two-stage FDR(S), we observe similar patterns, however the rejection region is not a rectangular form which means the rejection for p_{2i} depends on p_{1i} coming from the conditional distribution. To summarise differences between two methods, both methods have a trade-off relationship between the first and second tests, but the major difference lies in the shape of the rejection regions. As mentioned earlier, in the case of two-stage FDR(H), if a variable passes the first stage through p_{1i}\leq\gamma_{1} , the same cutoff value is applied to p_{2i} in the second stage. In this case, if it was eliminated in the first stage ( p_{1i}>\gamma_{1} ), there is no possibility of rejecting the primary variable regardless of the value of p_{2i} . On the other hand, in the case of two-stage FDR(S), the rejection region of p_{2i} gradually becomes larger depending on the significance strength shown by the auxiliary variable (small p_{1i} ). Unlike two-stage FDR(H), two-stage FDR(S) has a possibility that the primary variable can be rejected in all range of p_{1i} . Figure 3: p_{i} ’s for two-stage procedures (A),(B) of two-stage FDR(H) and (C) of two-stage FDR(S). In (A), p_{i}^{H} with the threshold \gamma_{1}=0.9 and (B) with the threshold \gamma_{1}=0.7 3.3 False discovery rate controlling procedures In the previous section, Theorem 3.1 and 3.2 show that suggested p -values for both methods are uniformly distributed when (\hat{\beta}_{i},y_{i})\sim G_{0}(\beta,y) . Based on this property of p_{i} s, we apply a method to control the FDR using p_{i}^{H} s and p_{i}^{S} s in ( 12 ) and ( 18 ). Our goal is to estimate the FDR and ensure that its value does not exceed a given FDR level, denoted as \alpha . We test M hypotheses in ( 5 ), for example, in the Set4 \Delta data introduced in section 2 , we test the significance of logfold changes . For a given \gamma , the estimated FDR is expressed as: \widehat{FDR({\gamma})}=\frac{\widehat{\pi_{0}}\gamma M}{\max(R_{M}(\gamma),1)} (19) where • M represents the number of hypotheses tested, • \widehat{\pi_{0}} is an estimate of the proportion of true negatives among M , • R_{M}(\gamma)=\#\{i:p_{i}\leq\gamma,1\leq i\leq M\} is the number of rejections, where p_{i} is either p_{i}^{H}(\gamma_{1}) or p_{i}^{S} . The denominator in ( 19 ) takes the maximum value between the number of rejected hypotheses and 1. Here, \widehat{\pi_{0}} is estimated by using \widehat{\pi_{0}}=\frac{\#\{p_{i}>\lambda\}}{(1-\lambda)M} (20) and we set \lambda=0.5 for this study. Selection of \gamma_{1} in two-stage FDR(H) Since p_{i}^{H}(\gamma_{1}) is uniformly distributed with any \gamma_{1}\in[0,1] , we select \gamma_{1} which rejects most to maximise the true positive rate. i.e. \displaystyle\widehat{\gamma} \displaystyle=\argmax_{\gamma}\big(\widehat{FDR({\gamma})}\leq\alpha\big), (21) \displaystyle\widehat{\gamma_{1}} \displaystyle=\argmax_{\gamma_{1}}\big(\#\{i:p_{i}^{H}(\gamma_{1})\leq\widehat{\gamma},1\leq i\leq M\}\big), (22) \displaystyle p_{i}^{H} \displaystyle=p_{i}^{H}(\hat{\gamma_{1}}). (23) We summarise our proposed methods controlling FDR in Algorithm 1 and Algorithm 2 . Algorithm 1 Two-stage FDR(H) 1: Estimate null distribution of \beta 2: Compute p_{2i} under the null distribution of \beta , and p_{1i} using the empirical distribution of Y 3: Select the copula between p_{1i} and p_{2i} 4: for \gamma_{1}\in\{r_{1},r_{2},\ldots,r_{K}\}\subset[0,1] do 5: Compute p_{i}^{H}(r_{k}) using p_{1i} , p_{2i} 6: Compute \widehat{\pi_{0}} 7: For fixed values of \alpha and \lambda , check the number of rejections ( m_{k} ) 8: end for 9: Compare the number of rejections for each r_{k} and choose \widehat{\gamma_{1}}=r_{k} as the one that maximises m_{k} 10: Reject if P_{i}^{H}(\widehat{\gamma_{1}})\leq\widehat{\gamma} , where \gamma is defined in ( 21 ) with \widehat{\gamma_{1}} Algorithm 2 Two-stage FDR(S) 1: Estimate null distribution of \beta 2: Compute p_{2i} under the null distribution of \beta , and p_{1i} using the empirical distribution of Y 3: Select the copula between p_{1i} and p_{2i} 4: Compute p_{i}^{S} using p_{1i} , p_{2i} with estimated copula 5: Compute \widehat{\pi_{0}} 6: Estimate \widehat{\gamma} satisfying ( 21 ) 7: Reject if p_{i}^{S}\leq\widehat{\gamma} 4 Simulation Studies In our study, we generate a dataset of M=8,000 pairs of variables, (\beta_{i},y_{i}) , for 1\leq i\leq 8,000 . This dataset is designed to simulate the characteristics of our real data example, the Set4 \Delta data, but in a simplified form. The auxiliary variable y_{i} \displaystyle y_{i}\sim Gamma(3,4) (24) with the mean value 3/4 and the primary variable \beta_{i} \displaystyle\beta_{i}\sim f(\beta) \displaystyle= \displaystyle p_{0}f_{0}(\beta)+(1-p_{0})f_{1}(\beta) \displaystyle= \displaystyle 0.95\phi(\beta)+0.05(0.5\phi(\beta-\mu)+0.5\phi(\beta+\mu)) for some \mu>0 . To closely replicate the joint distribution observed in the Set4 \Delta data, we model the pairs (\beta_{i},y_{i}) to be jointly distributed following a Clayton copula. In our first scenario we used the Clayton copula to calculate p values. We varied the parameters \mu and Kendall’s \tau and assessed the {FDR} in ( 9 ). Our Monte Carlo simulations provide an approximate FDR which is FDR\approx\frac{1}{K}\sum_{k=1}^{K}\frac{V_{k}}{R_{k}\vee 1} where R_{k} and V_{k} are the number of rejections and that of false rejections, respectively at k th simulation among K simulations. Let \theta_{i}={I}(H_{1i}~\mbox{is true}) . As statistical powers, we use the true positive rate (TPR) which is \displaystyle TPR=E\left(\frac{\sum_{i=1}^{M}\theta_{i}\delta_{i}}{\sum_{i=1}^{M}\theta_{i}\vee 1}\right) (25) where a\vee 1=\max(a,1) is the maximum of a and 1, \sum_{i=1}^{M}\theta_{i} is the number of H_{1i} and \sum_{i=1}^{M}\theta_{i}\delta_{i} is the number of the true positives. Similarly, an approximate TPR is calculated \frac{1}{K}\sum_{k=1}^{K}\frac{S_{k}}{M_{k}} where S_{k} and M_{k} are the number of true rejections and that of H_{1i} at k th simulation among K simulations. We use K=1000 in our simulation studies. We consider the various methods as follows : locfdr in 6 , Storey in 16 , IHW in 8 , Boca and Leek in 1 and our proposed two-stage FDR methods such as Type H and Type S. The results shown in Tables 1 show the effects of different degrees of dependence ( \tau ) with a constant distance between the null and alternative distributions ( \mu=3.0 ), and \mu changes while \tau remains fixed at -0.4. Notably, all methods indicate an increase in TPR as \mu increases. The one-stage FDR methods, which depend only on \hat{\beta}_{i} values, show consistent FDR and TPR, highlighting their limitations in adapting to different levels of dependence. Conversely, both covariate-assisted and two-stage FDR methods show higher TPR with increasing dependence. However, their effectiveness in controlling FDR varies. Covariate-assisted methods, despite extending the rejection region to reject more hypotheses, fail to maintain FDR control when dependencies are present. This suggests that covariate-assisted methods extend the rejection region where null hypotheses accumulate. Meanwhile, two-stage FDR methods effectively maintain FDR control even as rejection rates increase with increasing dependency, thereby achieving a better TPR. This result suggests that the inclusion of auxiliary variables can improve the discovery of significant results, but the results should be interpreted with caution. In Section 5, we demonstrate the potential for increased discovery using auxiliary variables. Still, the results of covariate-assisted methods should be treated with particular caution. Table 1: Comparison of False Discovery Rate (FDR) and True Positive Rate (TPR) across different settings for the mean parameter, \mu , and Kendall’s \tau coefficient. \tau One-stage FDR Covariate-assisted FDR Two-stage FDR locfdr Storey IHW Boca and Leek AdaFDR type H type S \widehat{FDR} 0.0 0.013 (0.013) 0.046 (0.020) 0.043 (0.020) 0.000 (0.006) 0.048 (0.024) 0.046 (0.021) 0.045 (0.020) -0.2 0.063 (0.032) 0.000 (0.006) 0.160 (0.061) 0.044 (0.019) 0.038 (0.016) -0.4 0.070 (0.039) 0.000 (0.007) 0.480 (0.085) 0.038 (0.016) 0.028 (0.012) -0.6 0.074 (0.041) 0.014 (0.020) 0.837 (0.019) 0.032 (0.018) 0.017 (0.009) -0.8 0.063 (0.045) 0.945 (0.002) 0.873 (0.007) 0.029 (0.028) 0.005 (0.006) \widehat{TPR} 0.0 0.213 (0.038) 0.378 (0.051) 0.367 (0.052) 0.002 (0.006) 0.375 (0.059) 0.379 (0.051) 0.376 (0.051) -0.2 0.364 (0.050) 0.004 (0.008) 0.380 (0.073) 0.490 (0.042) 0.577 (0.042) -0.4 0.364 (0.047) 0.014 (0.020) 0.500 (0.062) 0.643 (0.033) 0.804 (0.026) -0.6 0.392 (0.039) 0.115 (0.052) 0.656 (0.037) 0.739 (0.025) 0.928 (0.014) -0.8 0.466 (0.039) 0.582 (0.028) 0.672 (0.051) 0.790 (0.022) 0.980 (0.007) \mu One-stage FDR Covariate-assisted FDR Two-stage FDR locfdr Storey IHW Boca and Leek AdaFDR type H type S \widehat{FDR} 2.0 0.022 (0.079) 0.026 (0.065) 0.174 (0.084) 0.000 (0.000) 0.643 (0.247) 0.030 (0.022) 0.024 (0.016) 2.5 0.016 (0.026) 0.038 (0.030) 0.121 (0.057) 0.000 (0.011) 0.691 (0.101) 0.033 (0.018) 0.026 (0.012) 3.0 0.013 (0.013) 0.046 (0.020) 0.070 (0.039) 0.000 (0.007) 0.580 (0.085) 0.038 (0.016) 0.028 (0.012) 3.5 0.010 (0.008) 0.049 (0.017) 0.081 (0.047) 0.003 (0.007) 0.442 (0.088) 0.045 (0.018) 0.030 (0.012) 4.0 0.008 (0.006) 0.051 (0.016) 0.130 (0.053) 0.008 (0.008) 0.404 (0.080) 0.050 (0.018) 0.031 (0.012) \widehat{TPR} 2.0 0.010 (0.007) 0.015 (0.013) 0.073 (0.026) 0.000 (0.000) 0.183 (0.090) 0.190 (0.035) 0.308 (0.045) 2.5 0.065 (0.022) 0.127 (0.041) 0.168 (0.034) 0.001 (0.002) 0.348 (0.070) 0.420 (0.039) 0.595 (0.038) 3.0 0.213 (0.038) 0.378 (0.051) 0.364 (0.047) 0.014 (0.020) 0.500 (0.062) 0.643 (0.033) 0.804 (0.026) 3.5 0.428 (0.044) 0.642 (0.041) 0.614 (0.037) 0.175 (0.060) 0.681 (0.055) 0.803 (0.024) 0.920 (0.016) 4.0 0.643 (0.038) 0.828 (0.027) 0.781 (0.026) 0.454 (0.063) 0.838 (0.035) 0.902 (0.017) 0.973 (0.009) Our simulation results indicate that our proposed methods can control the False Discovery Rate (FDR) when the dependency structure is known. However, in real data, the copula is often unknown. To address the question, ”What happens if we mis-specify the copula in the models?” we explore the consequences of choosing an incorrect copula for calculating p_{i} in our proposed methods. While the correct copula selection is essential for controlling the FDR, it is equally crucial to understand the consequences of using an incorrect copula. To evaluate the robustness of our methodology to the selection of copula, we choose a copula that does not align with the true underlying dependencies. Specifically, we chose the BB7, BB6 and Joe copulas for comparison. These were chosen because of their similarity to the Clayton copula, which allowed us to explore the effects of potential misspecification when replacing the Clayton copula with these alternatives. Table 2 displays the estimated FDR and true positive rate (TPR) for each of the four copulas when \mu=3 and \tau=-0.4 . Table 2: Comparison of False Discovery Rate (FDR) and True Positive Rate (TPR) in the Presence of Misspecified Copulas. Copula One-stage FDR Two-stage FDR locfdr Storey type H type S \widehat{FDR} Clayton 0.013 (0.013) 0.046 (0.020) 0.038 (0.016) 0.028 (0.012) BB7 0.006 (0.012) 0.001 (0.0.02) BB6 0.008 (0.009) 0.002 (0.003) Joe 0.049 (0.019) 0.032 (0.014) \widehat{TPR} Clayton 0.213 (0.038) 0.378 (0.051) 0.643 (0.033) 0.804 (0.026) BB7 0.466 (0.035) 0.588 (0.030) BB6 0.518 (0.037) 0.630 (0.032) Joe 0.660 (0.032) 0.805 (0.026) Copula One-stage FDR Two-stage FDR locfdr Storey type H type S \widehat{FDR} Clayton 0.013 (0.013) 0.046 (0.020) 0.038 (0.016) 0.028 (0.012) BB7 0.038 (0.017) 0.029 (0.012) BB6 0.036 (0.016) 0.034 (0.013) Joe 0.031 (0.014) 0.028 (0.011) \widehat{TPR} Clayton 0.213 (0.038) 0.378 (0.051) 0.643 (0.033) 0.804 (0.026) BB7 0.643 (0.033) 0.805 (0.026) BB6 0.641 (0.033) 0.800 (0.026) Joe 0.643 (0.033) 0.804 (0.026) From Table 2 , we see that all four methods control a given level of FDR for all settings while locfdr and two-stage FDR (S) obtain lower FDR values compared to the nominal level \alpha=0.05 . However, two-stage FDR (S) and two-stage FDR (H) obtain highest and the second highest TPRs, respectively across all settings. These results show that the supplementary information from the auxiliary variable may affect the power of testing procedures when it is appropriately reflected in testing procedure. 5 Real Data Analysis 5.1 Estimation of the Marginal p -values and Optimal Copula auxiliary variable ( p_{1i} ) : The standard deviation of the logfold changes of each gene is used as an auxiliary variable. Since we have small number of samples, we estimated the standard deviation by bootstrap method for all possible combinations, resulting in 729 (i.e. 3^{3}\times 3^{3} ) possible combinations, and derived the p_{1i} values from their empirical distribution. The detailed proceudure is described in Algorithm 3 . Algorithm 3 Estimation of Standard Deviation using Bootstrap for Set4 \Delta data 1: for i\leftarrow 1 to 8220 do \triangleright Iterate through all genes 2: for b\leftarrow 1 to 729 do \triangleright Iterate through all bootstrap samples 3: Sample three knockouts ( KO_{i(1)}^{b},KO_{i(2)}^{b},KO_{i(3)}^{b} ) from the original knockouts ( KO_{i1},KO_{i2},KO_{i3} ) with replacement. 4: Sample three wildtypes ( WT_{i(1)}^{b},WT_{i(2)}^{b},WT_{i(3)}^{b} ) from the original wildtypes ( WT_{i1},WT_{i2},WT_{i3} ) with replacement. 5: Calculate logfold changes: \tilde{\beta}_{i}^{b}=\log\frac{\bar{KO}^{b}}{\bar{WT}^{b}} 6: end for 7: Compute the sample standard deviation of \tilde{\beta}_{i}^{b} values 8: Determine the p_{1i} value based on the empirical distribution of these standard deviations 9: end for primary variable ( p_{2i} ) : For the primary variable, we adopt the approach proposed by 12 to evaluate the log change values of each gene. This method estimates the null distribution as a mixture of two normal distributions.i.e. f_{0}(\beta)=0.615\phi\big(\frac{\beta}{0.063}\big)+0.385\phi\big(\frac{\beta+0.002}{0.205}\big) where \phi is a probability distribution function of a normal distribution. Based on this null, p_{2i} is calculated as described in ( 11 ). Optimal Copula : we evaluated the validity of various copulas - including Gaussian, Frank, Clayton, Gumbel and Joe - for modelling the dependencies between p_{1i} and p_{2i} . This evaluation, based on Log-Likelihood (LogLik), Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC), is illustrated in Table 3 . The Clayton copula was selected owing to its outstanding performance in all criterion, indicating an optimal balance between model complexity and fit. Table 3: The Log-likelihood(LogLik), AIC, BIC of Gaussian, Frank, Clayton, Gumbel and Joe copulas with set4 \Delta dataset Family LogLik AIC BIC Gaussian 306.45 -610.91 -604.80 Frank 332.84 -663.68 -657.57 Clayton 393.08 -784.16 -778.05 Gumbel 201.79 -401.57 -395.46 Joe 96.29 -190.57 -184.46 5.2 Application of FDR Controlling Procedure Based on the estimation of the distribution G_{0} in the previous section, we apply the Algorithm 1 and Algorithm 2 with \alpha=0.10 . For the distributions of p_{i}^{H} and p_{i}^{S} , see Figure in the supplementary material. In Algorithm 1 , the process of selecting \gamma_{1} involved analysing the relationship between the number of hypothesis rejections and various \gamma_{1} values. Using the equation ( 22 ), we determined the optimal value of \hat{\gamma}_{1} as the value that maximises the number of rejections. Figure 4 illustrates the relationship between \gamma_{1} values and the number of rejections. It shows that the number of rejections increases as \gamma_{1} increases, peaking at \gamma_{1}=0.987 , after which it starts to decrease. With the choice of \hat{\gamma}_{1}=0.987 , this resulted in the rejection of 492 null hypotheses out of a total of 8,220 . Figure 4: Selection of \gamma in two-stage FDR(H). The x-axis displays \gamma and the y-axis represents the corresponding number of rejection with two-stage FDR(H). In order to assess the performance of four different FDR controlling methods, we compare the number of rejections for both one-stage and two-stage approaches. Firstly, employing the one-stage FDR(locfdr), we identified 249 rejected genes. A slightly higher number of genes, 424 , were rejected when applying the one-stage FDR(Storey). However, when utilizing two-stage FDR methods, we observed a significant increase in gene rejections. Specifically, the two-stage FDR(H) resulted in 485 rejected genes, while the two-stage FDR(S) led to a 582 rejections. These findings support the simulation results presented in Table 1 , demonstrating the greater power of two-stage FDR methods in detecting deferentially expressed genes (DEG). Figure in online supplementary matierial offers an in-depth comparative analysis of the proposed two-stage FDR methods introduced in this study versus traditional one-stage FDR methods (local FDR and Storey’s method), applied to the set4 \Delta mutant dataset. Figure 5 shows four different rejection regions corresponding to the two one-stage FDR methods and the two two-stage FDR methods. These different rejection regions provide further insight into the comparison between one-stage and two-stage FDR methods. As shown in the figures, the one-stage FDR methods determine the rejection region based on only the logfold changes in the y -axis, while in the case of the two-stage FDR(H), it exhibits a rectangular-shaped rejection region determined independently by cutoff values for y and \beta . On the other hand, in the case of two-stage FDR(S), it shows a curved rejection region, indicating that the dependence between the two variables has been taken into account. The further result compared to DESeq2 ( 11 ), a well-known DEG analysis, is also represented in the supplementary material. HXT gene family : We conducted an investigation into the functional role of these newly rejected gene sets using GO and MIPs databases of the two rejected gene lists in 14 . In both cases, we observed an enrichment of genes associated with stress response and cell wall maintenance among the differentially expressed genes, consistent with our previous findings in 9 . Additionally, the new analysis revealed an interesting enrichment of genes involved in hexose transport, particularly from the HXT gene family. Figure 5 include the locations of the HXT gene family represented by crosses and show that the two-stage FDR(S) reject all of them while the others reject part of them. These rejected genes mainly belong to the minor category of hexose transporters, known for primarily transporting mannitol and sorbitol ( 3 ). Remarkably, under stress conditions like hypoxia or anaerobia, the transcription of these transporters is altered, likely leading to increased transport of the preferred carbon source, glucose, into cells ( 13 ). This finding identifies another class of genes regulated by Set4 during hypoxia that play a crucial role in the cell’s adaptation to low oxygen. Moreover, a number of these genes are located within approximately 40 kB of the chromosome end, including HXT15, HXT13, and HXT8, consistent with previous findings that Set4 regulates genes within the subtelomeric regions of chromosomes ( 9 ). In conclusion, this differential gene expression analysis method has successfully identified a new class of genes involved in stress responses, regulated by the Set4 protein in yeast grown under hypoxic conditions. Figure 5: Comparison of Rejection Regions: One-Stage vs. Two-Stage FDR Methods. The x-axis represents the standard deviation of logfold changes on a logarithmic scale as an auxiliary variable, while the y-axis shows the logfold changes as the primary variable. The sky-blue points corresponds to the rejection region at \alpha<0.05 , and the darker blue points represents the rejection region at \alpha<0.10 . In the legend, ✕ and ✴ signify the HXT gene family rejection with \alpha=0.05 and \alpha=0.10 , respectively, while ▲indicates non-rejected genes in the HXT gene family. Table 4: Comparison of total and HXT family gene rejections using one-stage (locfdr, Storey), covarate-assisted (IHW, Boca and Leek, AdaFDR) and two-stage (types H and S) FDR methods at \alpha=0.05 and \alpha=0.10 . One-stage FDR Covariate-assisted FDR Two-stage FDR \alpha locfdr Storey IHW Boca and Leek AdaFDR type H type S # rejected genes 0.05 230 361 486 110 844 407 503 0.10 249 424 633 116 1329 485 582 # of rejected HXT family 0.05 3 4 5 3 6 3 4 0.10 3 4 5 3 6 4 6 6 Concluding Remarks In this paper, we develop the two stage procedures controlling FDR based on incorporating an auxiliary variable to improve the power of testing the primary variable. Furthermore, we also increase the credibility of the results via incorporating prior knowledge through the auxiliary variable. Rather than assuming the structure of standard deviation of logfold changes, we simply used the copula to model the dependence between the logfold changes and their standard deviations. Our methods allow for a more flexible and accurate representation of the joint distribution of these two variables. Two proposed methods select the significant genes in different ways. One takes the form of screening, while the other manifests as a progressive form by showing different rejection regions based on the values of auxiliary variable. Both methods are superior to existing methods without using any auxiliary variable. By taking into account the variability of the experiments and controlling for false discovery rate, the proposed methods can help to increase the confidence and reproducibility of gene expression analysis. We present numerical studies supporting our proposed methods and the real data example of Set4 \Delta mutant data. Our proposed methods can control a given level of FDR and are more powerful in detecting the true alternative hypotheses than one-stage procedures and some existing methods with covariate. Especially in the real data, two-stage FDR (S) is more efficient to select all the genes in the HXT gene family compared to all the other methods. Overall, the proposed methods incorporating additional information into the problem of testing the primary variable are a valuable contribution to the field of gene expression analysis, and have the potential to improve the quality and reliability of experimental results. Additionally, the proposed methods can be widely applied beyond the gene-related data we introduced, to multiple testing problems where there exist both the primary and auxiliary variables. 7 Data and Code Availability The dataset(Set4 \Delta ) and source code developed for this research can be found at the following GitHub repository: https://github.com/twostageFDR/.github . This repository includes all scripts and instructions necessary to replicate the computational procedures and analyses presented in this paper. Conflict of Interest The authors have declared no conflict of interest. Supporting Information Additional Supporting Information may be found in the online version of this article.