Unsupervised Feature Selection Through Group Discovery

Shira Lifshitz, Ofir Lindenbaum, Gal Mishne, Ron Meir, Hadas Benisty · 2025

Read Unsupervised Feature Selection Through Group Discovery by Shira Lifshitz, Ofir Lindenbaum, Gal Mishne, Ron Meir, Hadas Benisty (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09166.txt.

Introduction

Shira Lifshitz and four co-authors argue that feature selection should judge groups of features, not single features alone. In unsupervised tasks without labels, useful signals often sit in related sets, like neighboring pixels or connected brain regions. Most existing methods still score each feature in isolation, or need groups fixed in advance.

Their method, GroupFS, learns the groups and selects the most informative ones in a single differentiable framework. It keeps feature and sample graphs smooth and uses a group sparsity penalty to favor compact structure. Across nine benchmarks covering images, tables, and biological data, GroupFS beats leading unsupervised methods at clustering. The groups it picks also line up with patterns people find meaningful.

Catalog record

CREATORShira Lifshitz, Ofir Lindenbaum, Gal Mishne, Ron Meir, Hadas Benisty
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-09166

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unsupervised Feature Selection Through Group Discovery Shira Lifshitz, Ofir Lindenbaum, Gal Mishne, Ron Meir, Hadas Benisty Abstract Unsupervised feature selection (FS) is essential for high-dimensional learning tasks where labels are not available. It helps reduce noise, improve generalization, and enhance interpretability. However, most existing unsupervised FS methods evaluate features in isolation, even though informative signals often emerge from groups of related features. For example, adjacent pixels, functionally connected brain regions, or correlated financial indicators tend to act together, making independent evaluation suboptimal. Although some methods attempt to capture group structure, they typically rely on predefined partitions or label supervision, limiting their applicability. We propose GroupFS, an end-to-end, fully differentiable framework that jointly discovers latent feature groups and selects the most informative groups among them, without relying on fixed a priori groups or label supervision. GroupFS enforces Laplacian smoothness on both feature and sample graphs and applies a group sparsity regularizer to learn a compact, structured representation. Across nine benchmarks spanning images, tabular data, and biological datasets, GroupFS consistently outperforms state-of-the-art unsupervised FS in clustering and selects groups of features that align with meaningful patterns. 1 Introduction Modern machine-learning systems routinely handle datasets with thousands to millions of features. Such high-dimensional data arise in neuroscience, finance, and computer vision ( 9 ; 8 ) . However, many of the observed features are nuisance, i.e., uninformative or noisy, and they obscure latent structure, inflate computational cost, and degrade generalization. Feature selection (FS) tackles this problem by retaining only the most relevant features, thereby discarding nuisance dimensions, reducing computational cost, and boosting downstream performance, e.g., clustering accuracy ( 11 ) . Because FS preserves the original measurements, the results remain interpretable and can enable domain-specific insights. In real-world applications such as neuroimaging, FS can lower acquisition costs by focusing on task-relevant regions, thus saving time or enabling higher resolution. Similarly, in domains like behavioral research, feature acquisition (e.g., questionnaires) can be expensive or burdensome, making efficient feature selection especially valuable. While many FS methods are supervised, even without labels, a well-chosen subset of features can uncover latent structure ( 24 ) . Yet selecting that subset is a complex combinatorial problem, and the challenge is amplified in the unsupervised setting where there are no labels to guide the selection process. Since obtaining annotations often requires costly expert effort, robust unsupervised FS is both challenging and essential ( 37 ; 23 ) . Classical FS methods can be categorized into three families. Filter methods assign scores to features using model‑agnostic criteria such as mutual information or graph smoothness ( 12 ; 3 ) . Wrapper methods search over subsets by repeatedly training a model, incurring high computational cost ( 19 ) . Embedded methods impose sparsity while training the model itself, e.g. LASSO ( 42 ) or stochastic‑gating networks ( 45 ; 39 ) . Most of these approaches, however, score features independently and ignore relationships among them. Real-world features often “act together”: spatially adjacent pixels, temporally co-varying sensors, or functionally coupled genes. Such relationships suggest that grouping features into meaningful subsets and selecting at the group level, rather than individually, can boost performance and provide clearer scientific insights. Existing group-aware FS methods either assume groups are known a priori or rely on supervision to form them ( 46 ; 13 ) . However, in many applications, the group structure is unknown, and fixing groups in advance can bias selection. Jointly discovering groups, selecting which ones are informative and rejecting the rest, without labels, remains an open problem, referred to as unsupervised group feature selection. In this paper, we address this gap by introducing GroupFS . It is a fully differentiable, end-to-end framework that simultaneously learns feature groups and selects the informative ones in a purely unsupervised manner. Our approach constructs two graphs: one over the sample space and another over the feature space, enforcing Laplacian smoothness on both. A feature-grouping and gating mechanism, guided by sparse regularization, dynamically discovers relevant feature groups. Our main contributions are as follows: • We introduce GroupFS, the first end-to-end, fully differentiable framework for unsupervised feature selection that jointly discovers latent feature groups and selects informative groups from them. • GroupFS automatically learns latent feature groups without relying on predefined partitions or supervision, thereby broadening its applicability to unlabeled, real-world data. • Extensive experiments on diverse synthetic and real-world datasets demonstrate that GroupFS consistently outperforms state-of-the-art unsupervised FS baselines in clustering accuracy and identifies meaningful feature groups. 2 Related Work Unsupervised FS. One line of research addresses the unsupervised FS problem by constructing a sample graph and selecting features that vary smoothly over the data manifold ( 12 ; 5 ; 26 ; 30 ; 27 ) . Autoencoder-based methods offer an alternative, ranking features by their contribution to reconstruction loss ( 1 ; 41 ) . However, reconstruction objectives do not necessarily promote features that capture the relationships among samples, which are essential for downstream tasks such as clustering. Group FS. Other work seeks to exploit feature groups, but most approaches assume the groups are fixed a priori, using heuristics or domain knowledge to define them ( 46 ; 47 ; 34 ; 44 ; 32 ) . Although effective in special cases, predefined groups limit adaptability and can introduce bias. A more flexible strategy is to learn groups during training. 13 take a step in this direction by jointly inferring group structure and training a classifier, but they rely on label supervision. 40 proposes a spectral approach to select groups of features; however, they assume a setting of differentiating between two or more given datasets. In contrast to these, we jointly discover feature groups and select the informative ones without any supervision, allowing structure to emerge directly from the data. 3 Preliminaries 3.1 Graphs and Spectral Analysis Let X=[\mathbf{x}_{1},\ldots,\mathbf{x}_{N}]^{\top}\in\mathbb{R}^{N\times d} be a data matrix, where row i (sample i ) is \mathbf{x}_{i}=X_{i:}\in\mathbb{R}^{d} , and column k (feature k ) is \mathbf{x}^{(k)}=X_{:k}\in\mathbb{R}^{N} . We assume the data lies on a low-dimensional manifold and capture its local geometry using an undirected, weighted graph G=(V,E,W) ( 43 ; 31 ) . Pairwise affinities are defined using the self-tuning kernel ( 48 ) as W_{ij}=\exp\!\left(-\frac{\|\mathbf{x}_{i}-\mathbf{x}_{j}\|_{2}^{2}}{\gamma_{i}\gamma_{j}}\right), (1) where \gamma_{i} is the distance from \mathbf{x}_{i} to its K -th nearest neighbor. This sample-dependent scaling adapts to local density, improving robustness in heterogeneous data. The degree matrix is defined as D=\operatorname{diag}(d_{1},\dots,d_{N}) , where d_{i}=\sum_{j}W_{ij} . Two standard graph operators are the normalized graph Laplacian L_{\text{sym}}=I-D^{-1/2}WD^{-1/2} and the random walk matrix P=D^{-1}W . Though both are used in spectral analysis, they differ in interpretation: eigenvectors corresponding to low eigenvalues of L_{\text{sym}} capture smooth, low-frequency variations on the graph, while those associated with high eigenvalues of P capture similar directions. The matrix power P^{t} represents transition probabilities after t steps of the random walk ( 38 ) . Laplacian Score (LS). 12 leveraged a sample graph structure for feature selection by ranking individual features based on their alignment with the graph’s smoothest modes. Features whose values vary minimally across strongly connected samples (i.e., along high‑weight edges) are preferred. For a feature vector \mathbf{x}^{(k)}\in\mathbb{R}^{N} , the Laplacian Score is \mathrm{LS}(\mathbf{x}^{(k)})\;=\;{(\mathbf{x}^{(k)})}^{\top}L_{\mathrm{sym}}\mathbf{x}^{(k)}\;=\;\sum_{i=1}^{N}\lambda_{i}\,\langle\mathbf{v}_{i},\mathbf{x}^{(k)}\rangle^{2}, where \{(\lambda_{i},\mathbf{v}_{i})\}_{i=1}^{N} are the eigenpairs of L_{\mathrm{sym}} . A smaller score indicates that the feature varies

Read the complete text