Sparse minimum Redundancy Maximum Relevance for feature selection

Peter Naylor, Benjamin Poignard, Héctor Climente-González, Makoto Yamada · 2025

Read Sparse minimum Redundancy Maximum Relevance for feature selection by Peter Naylor, Benjamin Poignard, Héctor Climente-González, Makoto Yamada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18901.txt.

Introduction

Peter Naylor, Benjamin Poignard, Héctor Climente-González, and Makoto Yamada published a 2025 paper on choosing which measurements matter in a dataset. Good feature selection keeps variables that say a lot about the target while dropping ones that merely repeat each other. Keeping useless variables adds noise and cost.

Their method turns that balance into a penalized score, where useless variables are driven exactly to zero. They prove conditions under which the true useless set is recovered. A multi-stage filter then controls the rate of false discoveries, so users set only an error threshold rather than a number of features. Tests on simulated and real data matched a leading rival method while keeping fewer features.

Catalog record

CREATORPeter Naylor, Benjamin Poignard, Héctor Climente-González, Makoto Yamada
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE109 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18901

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparse minimum Redundancy Maximum Relevance for feature selection Peter Naylor, Benjamin Poignard, Héctor Climente-González, Makoto Yamada Abstract We propose a feature screening method that integrates both feature-feature and feature-target relationships. Inactive features are identified via a penalized minimum Redundancy Maximum Relevance (mRMR) procedure, which is the continuous version of the classic mRMR penalized by a non-convex regularizer, and where the parameters estimated as zero coefficients represent the set of inactive features. We establish the conditions under which zero coefficients are correctly identified to guarantee accurate recovery of inactive features. We introduce a multi-stage procedure based on the knockoff filter enabling the penalized mRMR to discard inactive features while controlling the false discovery rate (FDR). Our method performs comparably to HSIC-LASSO but is more conservative in the number of selected features. It only requires setting an FDR threshold, rather than specifying the number of features to retain. The effectiveness of the method is illustrated through simulations and real-world datasets. The code to reproduce this work is available on the following GitHub: https://github.com/PeterJackNaylor/SmRMR . Keywords: FDR, Feature screening, mRMR, Redundancy, Sparsistency 1 Introduction A much employed approach to solve high-dimensional problems is sparse modelling, which focuses on variable selection by discarding unimportant variables for prediction. It aims to recover the true signal when the underlying model admits a sparse representation by applying a penalization procedure to the objective function. A flourishing literature has been dedicated to the devise of suitable penalty functions that can best fit the observed patterns and that satisfy desirable properties: in particular, see Fan and Li 2001 and Fan and Peng 2004 for asymptotic results; see Loh and Wainwright 2017 or Poignard and Fermanian 2022 for a non-asymptotic analysis. When the data is ultra high-dimensional, that is p\gg n with p the number of variables and n the sample size, sparsity-based methods may not perform well due to their computational cost, lack in both statistical accuracy and algorithmic stability. In light of these issues, Fan and Lv 2008 developed the sure independent screening (SIS) approach. The latter is based on marginal Pearson correlation learning between the features and the target variable. However, the procedure relies on a linear model with Gaussian predictors and responses so that it may not be robust to model misspecification. This gave rise to a broad range of studies on model-free SIS methods: the Distance Covariance of Székely et al. 2007 and Székely and Rizzo 2009 ; the Distance Correlation of Li et al. 2012 ; the \sup -HSIC of Balasubramanian et al. 2013 ; the Kolmogorov filter of Mai and Zou 2013 and Mai and Zou 2015 ; the Projection Correlation of Liu et al. 2022 ; and the mixture Kendall’s Tau and Spearman Rho of Poignard et al. 2022 , among others. As pointed out by Mai and Zou 2015 , feature screening and sparsity-based selection methods differ in their objectives: on the one hand, variable selection attempts to recover the true sparse signal, which may require high-level conditions such as the incoherence assumption; whereas on the other hand, feature screening aims to discover a majority of inactive features and it is thus less ambitious and requires weaker assumptions. Furthermore, the technique employed to discover the features truly associated with the target variable should result in a limited number of false discoveries, that is, the False Discovery Rate (FDR) should be low. The selected model should be as large as possible while controlling the FDR, that is the type S error introduced by Gelman and Tuerlinckx 2000 . To satisfy this constraint in linear regression analysis, Barber and Candès 2015 developed the knockoff filter, which creates copies of the original variables serving as control variables to guarantee the control of the FDR by the feature selection technique. In addition to upper bounding the directional FDR as well as the standard FDR, it allows to report the selected features for a given confidence level. This novel technique has been extended in Barber and Candès 2019 to accommodate the p\gg n case and to deal with the model-free viewpoint, as in, e.g., Romano et al. 2020 . In particular, the pertinence of the method has been illustrated in the context of biological data: Srinivasan et al. 2021 proposed a two-step compositional knockoff filter to control the FDR in microbiome data; Tian et al. 2022 developed Grace-AKO for graph-constrained estimation and improved the feature selection stability of gene selection by repeating the knockoff procedure and aggregating information. In this paper, we propose a two step approach for feature screening. After screening out the inactive features, the second step aims to upper bound the FDR at a pre-specified level. In contrast with the standard SIS procedure, which only considers the feature-target variable relationship, our approach refines the screening step by incorporating the feature-feature relationship. This procedure builds upon the minimum redundancy maximum relevance (mRMR) of Peng et al. 2005 and the HSIC-LASSO of Yamada et al. 2014 to solve the redundancy issue. More precisely, we consider the following problem: given n observations of p features X_{1},\ldots,X_{p} , we screen out the features whose marginal utility with the target variable Y is not significant. To do so, we develop a novel screening procedure, called sparse mRMR (SmRMR hereafter), which extends the standard mRMR algorithm through the introduction of a non-convex penalization. Indeed, the SmRMR criterion can be recast as a penalized M-estimation problem, where the mRMR parameters and their zero entries correspond to the relaxation parameters of the original mRMR problem and to non-significant dependence, respectively. The optimal threshold selection for screening out the features with the SmRMR is performed via cross-validation, contrary to model-free screening procedures. In the second step, we apply the knockoff filter to the features selected in the first step, allowing us to report the selected features for a given confidence level. Our main contributions are as follows: we establish the existence of a consistent estimator for the SmRMR problem and show that the true zero elements of the mRMR criterion can be consistently estimated, ensuring the correct identification of the non-active features; conditional on the screening step, using the knockoff filter on the selected features, we verify that our SmRMR method controls the FDR for a given confidence level. The remainder of the paper is organized as follows. Section 2 details the framework and the SmRMR procedure. Section 3 is devoted to the theoretical properties of the SmRMR. Section 4 details the knockoff filter for SmRMR and FDR control. Section 5 illustrates the performances of the proposed method through simulations and real data experiments. All the proofs of the main text and auxiliary results are relegated to the Appendix. Notations. Throughout this paper, we denote the cardinality of a set E by \text{card}(E) . For a vector \mbox{\boldmath$v$}\in{\mathbb{R}}^{d} , the \ell_{p} norm is \|\mbox{\boldmath$v$}\|_{p}=\big(\sum^{\text{d}}_{k=1}|\mbox{\boldmath$v$}_{k}|^{p}\big)^{1/p} for p>0 , and \|\mbox{\boldmath$v$}\|_{\infty}=\max_{i}|\mbox{\boldmath$v$}_{i}| . For a symmetric matrix A , \lambda_{\min}(A) (resp. \lambda_{\max}(A) ) denotes the minimum (resp. maximum) eigenvalue of A , and \text{tr}(A) is the trace operator. For a matrix B , \|B\|_{s}=\lambda^{1/2}_{\max}(B^{\top}B)^{1/2} and \|B\|_{F}=\text{tr}(B^{\top}B) are the spectral and Frobenius norms, respectively, and \|B\|_{\max}=\max_{ij}|B_{i,j}| is the coordinate-wise maximum. For a function f:{\mathbb{R}}^{d}\rightarrow{\mathbb{R}} , we denote by \partial f the component-by-component partial derivative of f

Read the complete text