Enhancing Signal Proportion Estimation Through Leveraging Arbitrary Covariance Structures

Jingtian Bai, Xinge Jessie Jeng · 2025

Read Enhancing Signal Proportion Estimation Through Leveraging Arbitrary Covariance Structures by Jingtian Bai, Xinge Jessie Jeng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-11922.txt.

Introduction

Jingtian Bai and Xinge Jeng improve estimates of how many true signals hide in a large set of variables, in a research paper from 2025. Such estimates help scientists judge how much of a finding is real. Older tools assume the variables are independent, which often fails in real data where variables move together.

Their new estimator uses the pattern of links between variables, whatever that pattern is. Theory shows how signal strength, sparseness, and these links interact. Tests with simulated data show the method is more accurate than leading rivals. It also picks up weaker signals that older tools would miss.

Catalog record

CREATORJingtian Bai, Xinge Jessie Jeng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE131 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-11922

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Signal Proportion Estimation Through Leveraging Arbitrary Covariance Structures Jingtian Bai, Xinge Jessie Jeng Abstract Accurately estimating the proportion of true signals among a large number of variables is crucial for enhancing the precision and reliability of scientific research. Traditional signal proportion estimators often assume independence among variables and specific signal sparsity conditions, limiting their applicability in real-world scenarios where such assumptions may not hold. This paper introduces a novel signal proportion estimator that leverages arbitrary covariance dependence information among variables, thereby improving performance across a wide range of sparsity levels and dependence structures. Building on previous work that provides lower confidence bounds for signal proportions, we extend this approach by incorporating the principal factor approximation procedure to account for variable dependence. Our theoretical insights offer a deeper understanding of how signal sparsity, signal intensity, and covariance dependence interact. By comparing the conditions for estimation consistency before and after dependence adjustment, we highlight the advantages of integrating dependence information across different contexts. This theoretical foundation not only validates the effectiveness of the new estimator but also guides its practical application, ensuring reliable use in diverse scenarios. Through extensive simulations, we demonstrate that our method outperforms state-of-the-art estimators in both estimation accuracy and the detection of weaker signals that might otherwise go undetected. Keywords : Adaptive estimation, Estimation consistency, Phase diagram, Lower bound estimators, Principal factor approximation, Sparse inference 1 Introduction Estimating the proportion of signal variables among a large number of noise variables is a fundamental problem in many scientific fields, including genomics, neuroscience, and environmental science. In these applications, datasets often contain a mix of truly informative variables (signals) and irrelevant ones (noise). Accurate estimation of the signal proportion provides crucial insights into the underlying structure of the data and informs the choice and calibration of statistical models, ultimately leading to more robust scientific conclusions. In this paper, we consider the multivariate normal model (X_{1},\cdots,X_{m})^{T}\sim N(\bm{\mu},{\bm{\Sigma}}), (1) where \bm{\mu}=(\mu_{1},\cdots,\mu_{m})^{T} is a sparse mean vector, with \mu_{i}=0 indicating a noise variable and \mu_{i}\neq 0 indicating a signal. The covariance matrix \bm{\Sigma} is assumed to be an arbitrary correlation matrix. This flexible model has been widely adopted for its generality and relevance in high-dimensional data analysis. Let I_{0}=\{i\in\{1,\cdots,m\}:\mu_{i}=0\} and I_{1}=\{i\in\{1,\cdots,m\}:\mu_{i}\neq 0\} denote the index sets of noise and signal variables, respectively. Our objective is to estimate the signal proportion, \pi=|I_{1}|/m. This estimation problem poses several key challenges. First, the distributions of signal variables are only partially specified due to the unknown nonzero means. Second, the performance of estimators can vary substantially with the unknown signal proportion \pi ( Meinshausen and Rice 2006 ) . Third, arbitrary dependencies in \bm{\Sigma} can severely degrade the performance of methods designed under independence assumptions ( Jeng 2023 ) . A variety of signal proportion estimators have been proposed, particularly in the context of multiple testing. Foundational work in Storey 2002 ; Genovese and Wasserman 2004 ; Meinshausen and Rice 2006 ; Jin and Cai 2007 ; Jin 2008 demonstrated that accurate estimation of \pi can significantly improve the performance multiple testing procedures. For a comprehensive review, Chen 2019 categorizes estimators by methodology, model assumptions, consistency properties, and practical limitations. In recent years, the problem of signal proportion estimation has gained renewed attention in the context of high-dimensional false negative control (HD-FNC), which targets the discovery of weak signals. In HD-FNC procedures, accurate estimation of \pi is a crucial step for effectively retaining signals that would otherwise be missed Jeng et al. 2016 ; Jeng and Chen 2019 ; Jeng et al. 2019 . This has motivated the development of methods that are adaptive to both unknown signal sparsity levels and general dependence structures Jeng et al. 2022 ; Jeng 2023 . Although signal proportion estimation is conceptually related to variable selection, the two problems are methodologically distinct. A variable selection procedure that consistently identifies the true set of signal variables can naturally yield an estimate of \pi by counting the selected signals. However, achieving selection consistency typically requires strong assumptions, such as a minimum signal strength or strict sparsity conditions. In contrast, consistent estimation of the signal proportion can often be achieved under much weaker assumptions, without requiring perfect separation between signal and noise variables. This fundamental distinction is well illustrated by the phase diagram framework in high-dimensional inference; see, for example ( Donoho and Jin 2004 ; Cai et al. 2011 ; Arias-Castro et al. 2011 ; Ji and Jin 2012 ; Cai and Sun 2017 ; Arias-Castro and Chen 2017 ; Gao and Stoev 2021 ; Donoho and Jin 2015 ; Ji and Zhao 2014 ; Jin et al. 2017 ; Chen et al. 2019 ; Chen et al. 2023 ) . Among existing proportion estimation approaches, methods that provide lower confidence bounds for \pi are especially useful when conservative estimates are needed. Meinshausen and Rice 2006 introduced lower 100(1-\alpha)\% confidence bounds using empirical processes of p -values under independence. Later work, including Blanchard et al. 2020 , Katsevich and Ramdas 2020 , and Jeng 2023 , extended these results to dependent settings, enhancing applicability. In particular, Jeng 2023 developed an omnibus lower bound estimator that adapts to a wide range of dependence structures while preserving lower bound guarantees, and showed through theoretical and empirical results that it is more robust and efficient than existing alternatives. Despite these advances, existing adaptive methods often do not fully leverage known dependence structures. In many real-world applications, prior knowledge about dependence is available and can be exploited for improved estimation. Examples include: • Gene expression studies, where known biological pathways inform gene-gene correlations; • Genome-wide association studies (GWAS), where linkage disequilibrium patterns define genetic correlations; • Functional MRI data, where brain connectivity maps inform spatial dependence; • Climate models, where physical laws and geography dictate spatio-temporal dependencies. In these scenarios, incorporating known dependence structures can significantly enhance the accuracy and reliability of signal proportion estimation. In this paper, we propose a new estimator for \pi that fully utilizes known dependence structures while maintaining consistency across a broad range of signal sparsity levels. To retain the lower bound property, we develop two complementary approaches: one that strictly guarantees a lower bound at the cost of higher computational burden, and another that is computationally efficient and asymptotically conservative under mild conditions. Both approaches integrate the principal factor approximation (PFA) of Fan et al. 2012 to enhance the signal-to-noise ratio by leveraging dependence information, resulting in substantial power gains. We present a theoretical analysis that examines how signal sparsity, signal strength, and covariance dependence jointly affect the performance of signal proportion estimation. To facilitate this analysis, we extend the phase diagram framework, which was originally developed to characterize the fundamental limits of signal detection and classification in high-dimensional inference ( Donoho and Jin 2004 ; Arias-Castro et al. 2011 ; Ji and Jin 2012 ; Cai and Sun 2017 ; Chen et al. 2019 ; Gao and Stoev 2021 ) , to the context of signal proportion estimation. This extension yields new insights into the regions where consistent estimation is achievable and where leveraging dependence information can provide significant advantages. Building on the new insight, we introduce a novel analytic tool that quantifies the potential benefit of dependence adjustment,

Read the complete text