Bi-Level Unsupervised Feature Selection

Jingjing Liu, Xiansen Ju, Xianchao Xiu, Wanquan Liu · 2025

Read Bi-Level Unsupervised Feature Selection by Jingjing Liu, Xiansen Ju, Xianchao Xiu, Wanquan Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20563.txt.

Introduction

Jingjing Liu, Xiansen Ju, Xianchao Xiu, and Wanquan Liu propose a two-level method for picking the most useful features from data with no labels, in this 2025 arXiv paper. Feature selection trims data down to what matters, but most label-free methods judge features from a single angle. They can miss either a feature's importance or the data's inner layout.

Their method works on two levels at once. A clustering level sketches the data's structure with stand-in labels, while a feature level applies a strict counting rule that forces whole features in or out. The authors solve the combined model with an efficient alternating algorithm and prove it settles down. Tests on two made-up and eight real datasets beat rival methods on clustering and sorting tasks.

Catalog record

CREATORJingjing Liu, Xiansen Ju, Xianchao Xiu, Wanquan Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bi-Level Unsupervised Feature Selection Jingjing Liu, Xiansen Ju, Xianchao Xiu, Wanquan Liu Abstract Unsupervised feature selection (UFS) is an important task in data engineering. However, most UFS methods construct models from a single perspective and often fail to simultaneously evaluate feature importance and preserve their inherent data structure, thus limiting their performance. To address this challenge, we propose a novel bi-level unsupervised feature selection (BLUFS) method, including a clustering level and a feature level. Specifically, at the clustering level, spectral clustering is used to generate pseudo-labels for representing the data structure, while a continuous linear regression model is developed to learn the projection matrix. At the feature level, the ℓ 2 , 0 \ell_{2,0} -norm constraint is imposed on the projection matrix for more effectively selecting features. To the best of our knowledge, this is the first work to combine a bi-level framework with the ℓ 2 , 0 \ell_{2,0} -norm. To solve the proposed bi-level model, we design an efficient proximal alternating minimization (PAM) algorithm, whose subproblems either have explicit solutions or can be computed by fast solvers. Furthermore, we establish the convergence result and computational complexity. Finally, extensive experiments on two synthetic datasets and eight real datasets demonstrate the superiority of BLUFS in clustering and classification tasks. I Introduction Recently, the rapid development of information technology has led to a significant increase in the number of instances and feature dimensions in various application datasets, especially in the fields of image processing and text classification, see [ 1 , 2 , 3 , 4 , 5 ] . Unsupervised feature selection (UFS), as an effective dimensionality reduction technique, has attracted much attention due to its independence from label information and good interpretability [ 6 , 7 , 8 ] . Nowadays, UFS has been successfully served as an effective preprocessing step for downstream tasks covering clustering and classification [ 9 ] . In high-dimensional data, UFS aims to identify a subset of features that retains the most discriminative information while preserving the intrinsic data structure [ 10 , 11 ] . Generally speaking, existing UFS methods can be categorized into three types: filtering-based, wrapper-based, and embedded-based. The representative UFS methods include Laplacian score (LapScore) [ 12 ] , multi-cluster feature selection (MCFS) [ 13 ] , unsupervised discriminative feature selection (UDFS) [ 14 ] , structured optimal graph feature selection (SOGFS) [ 15 ] , and robust neighborhood embedding (RNE) [ 16 ] . Notably, embedded-based methods integrate structure learning and feature selection within a unified framework, thereby combining the strengths of both filtering- and wrapper-based methods. Fig. 1: Comparisons of BLSFS [ 17 ] and our proposed BLUFS. BLSFS is composed of classification and feature levels. In the classification level, discrete pseudo-labels are obtained through spectral clustering, and then a projection matrix is learned via classifier training. In the feature level, a similarity matrix is obtained using an adaptive graph learning model. Finally, by applying the \ell_{2,1} -norm to W , features with higher rankings are obtained. BLUFS is composed of clustering and feature levels. Unlike BLSFS, in the clustering level, continuous pseudo-labels representing the clustering structure are obtained through spectral clustering, and a projection matrix is obtained via a linear transformation model. In the feature level, after obtaining the similarity matrix, the number of features is strictly limited and sufficient sparsity is ensured by imposing the \ell_{2,0} -norm constraint on W . Principal component analysis (PCA) and its variants are widely recognized in the UFS domain for their simplicity and computational efficiency [ 18 , 19 ] . Recent studies have focused on enhancing PCA-based methods by incorporating sparsity constraints, which significantly improves their feature selection capability [ 20 , 21 ] . For example, Yang et al. [ 14 ] proposed a sparse PCA model by introducing the \ell_{2,1} -norm regularization [ 22 ] , which promotes group sparsity in the projection matrix. Li et al. [ 23 ] introduced the \ell_{2,p} -norm with p\in(0,1) [ 24 ] and reformulated PCA as a reconstruction error minimization problem, which is denoted as SPCAFS. To improve the interpretability, Nie et al. [ 25 ] proposed FSPCA, a feature sparse PCA model that employs the \ell_{2,0} -norm [ 26 ] to enforce row sparsity in the projection matrix, enabling consistent feature selection across multiple components. In a related work, Nie et al. [ 27 ] designed a coordinate descent framework with provable convergence guarantees. Zhou et al. [ 28 ] further introduced an evolutionary sparsity-based method to enhance the flexibility and performance. In fact, the \ell_{2,0} -norm has been applied in many data engineering fields. For example, Chen et al. [ 29 ] constructed \ell_{2,0} -norm-based multivariate generalized linear models; Zhang et al. [ 30 ] introduced the \ell_{2,0} -norm and non-negative matrix factorization for fault detection; Zhu et al. [ 31 ] applied the \ell_{2,0} -norm to subspace clustering; Huang et al. [ 32 ] proposed the \ell_{2,0} -norm penalized graph trend filtering model for estimating piecewise smooth signals. In fact, compared with the \ell_{2,1} -norm and the \ell_{2,p} -norm with p\in(0,1) , the \ell_{2,0} -norm directly enforces group sparsity without requiring intricate parameter tuning to balance sparsity and accuracy [ 33 , 34 ] . This property allows it to preserve key data features and improve feature-level interpretability [ 35 ] . The aforementioned methods conduct feature selection solely at the feature level, without considering downstream tasks such as clustering or classification. However, in many real-world scenarios, feature selection is a hierarchical decision-making process [ 36 ] . Therefore, adopting a hierarchical structure to model dependencies between multiple tasks is particularly important for feature selection. Recently, researchers have begun to explore feature selection by integrating information from different levels. Zhou et al. [ 37 ] attempted to perform feature selection by integrating multiple feature selection methods at different levels, which is called is BLSEF. At the clustering level, multiple clustering results are generated using various algorithms and then aggregated via self-paced learning to form a consensus that subsequently guides feature selection at the feature level. At the feature level, multiple base feature selection methods are integrated to learn a consensus feature score. Hu et al. [ 17 ] tried to perform feature selection at both the classification level and the feature level, so that the selected features can be suitable for different downstream tasks. Specifically, spectral clustering is employed at the classification level to generate pseudo-labels that guide feature selection. Meanwhile, adaptive graph learning is applied at the feature level, followed by a ranking procedure to select the most informative features. This method is abbreviated as BLSFS. It is worth noting that although the above bi-level UFS methods show good performance, there exist two shortcomings. On the one hand, the pseudo-labels in spectral clustering are too strict, and on the other hand, the \ell_{2,0} -norm has not yet been used to directly achieve feature sparsity. Inspired by the aforementioned observations, we propose a bi-level unsupervised feature selection method, called BLUFS, to distinguish it from existing methods. In our proposed BLUFS, spectral clustering is employed to generate a more flexible scaled cluster indicator matrix. Unlike BLSFS, the constraints are relaxed to construct a continuous matrix representation, thus avoiding

Read the complete text