Continuous Optimization for Feature Selection with Permutation-Invariant Embedding and Policy-Guided Search

Rui Liu, Rui Xie, Zijun Yao, Yanjie Fu, Dongjie Wang † · 2025

Read Continuous Optimization for Feature Selection with Permutation-Invariant Embedding and Policy-Guided Search by Rui Liu, Rui Xie, Zijun Yao, Yanjie Fu, Dongjie Wang † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11601.txt.

Introduction

Rui Liu and four co-authors search feature subsets in a smooth space in a 2025 paper. Feature selection trims redundant inputs for faster, clearer models. The best subset depends on tangled interactions among features. Generative methods search a learned continuous embedding of subsets instead.

Order sensitivity poisons such embeddings, since a set has no natural sequence. Their encoder builds subset vectors from pairwise feature relations, erasing order effects. Gradient search in that space can stall on bumpy, non convex ground. A policy guided search explores without assuming convexity. Inducing points speed up the pairwise computation. The framework adapts across scenarios better than fixed heuristics. Cleaner inputs follow for any downstream model.

Catalog record

CREATORRui Liu, Rui Xie, Zijun Yao, Yanjie Fu, Dongjie Wang †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11601

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Continuous Optimization for Feature Selection with Permutation-Invariant Embedding and Policy-Guided Search Rui Liu, Rui Xie, Zijun Yao, Yanjie Fu, Dongjie Wang † Abstract . Feature selection removes redundant features to enhance both performance and computational efficiency in downstream tasks. Existing methods often struggle to capture complex feature interactions and adapt to diverse scenarios. Recent advances in this domain have incorporated generative intelligence to address these drawbacks by uncovering intricate relationships between features. However, two key limitations remain: 1) embedding feature subsets in a continuous space is challenging due to permutation sensitivity, as changes in feature order can introduce biases and weaken the embedding learning process; 2) gradient-based search in the embedding space assumes convexity, which is rarely guaranteed, leading to reduced search effectiveness and suboptimal subsets. To address these limitations, we propose a new framework that can: 1) preserve feature subset knowledge in a continuous embedding space while ensuring permutation invariance; 2) effectively explore the embedding space without relying on strong convex assumptions. For the first objective, we develop an encoder-decoder paradigm to preserve feature selection knowledge into a continuous embedding space. This paradigm captures feature interactions through pairwise relationships within the subset, removing the influence of feature order on the embedding. Moreover, an inducing point mechanism is introduced to accelerate pairwise relationship computations. For the second objective, we employ a policy-based reinforcement learning (RL) approach to guide the exploration of the embedding space. The RL agent effectively navigates the space by balancing multiple objectives. By prioritizing high-potential regions adaptively and eliminating the reliance on convexity assumptions, this search strategy effectively reduces the risk of converging to local optima. Finally, we conduct extensive experiments to demonstrate the effectiveness, efficiency, robustness and explicitness of our model. Our code and dataset are publicly accessible on GitHub 1 1 1 https://github.com/RayLiu1103/CAPS . Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25), August 3–7, 2025, Toronto, ON, Canada DOI: 10.1145/3711896.3736891 ISBN: 979-8-4007-1454-2/2025/08 CCS: Computing methodologies Feature selection † † cc-license: by 1. Introduction Feature selection eliminates redundant and irrelevant features to improve both predictive performance and computational efficiency in downstream tasks. Despite the growing dominance of deep learning, feature selection remains indispensable in scenarios characterized by high-dimensional data, the need for interpretability, and limited resource constraints. For instance, in healthcare, identifying key biomarkers from extensive genetic datasets enhances model transparency and supports informed decision-making. In finance, extracting relevant features from transactional data facilitates the development of efficient and interpretable risk models. These examples demonstrate the importance of feature selection in addressing real-world challenges, even in the era of deep learning. Existing works can be divided into three categories: 1) filter methods ( Yang and Pedersen, 1997 ; Forman and others, 2003 ; Hall, 1999 ; Yu and Liu, 2003 ) rank features based on specific scoring criteria. They often select the top-K ranking features as the optimal subset; 2) wrapper methods ( Yang and Honavar, 1998 ; Kim et al., 2000 ; Narendra and Fukunaga, 1977 ; Kohavi and John, 1997 ) iteratively generate and evaluate feature subsets. They select the feature subset that performs best on a downstream task as the optimal one; 3) embedded methods ( Tibshirani, 1996 ; Sugumaran et al., 2007 ) integrate a feature selection regularization term (e.g., L_{1} regularization) into model training. As the model converges, irrelevant features are penalized, and the non-zero coefficients indicate the optimal subset. While prior work has achieved notable success, it falls short in capturing complex feature interactions and adapting feature selection knowledge to diverse and dynamic scenarios. Recently, the success of generative AI has inspired researchers to revisit the feature selection problem through a generative lens. By embedding discrete feature selection knowledge into a continuous embedding space, they not only capture complex feature interactions for generating enhanced feature subsets but also make the knowledge more adaptable to diverse scenarios ( Xiao et al., 2023c ; Ying et al., 2024a ; Ying et al., 2023 ; Ying et al., 2024b ; Azim et al., 2024 ; Hu et al., 2024 ) . However, there are two primary limitations: (1) Limitations 1: Permutation bias bring noise into the embedding space and lead to suboptimal performance. Existing methods fail to encode the fact that feature order does not impact model performance within the learned feature subset embedding. This oversight introduces bias into the embedding space, limiting the effectiveness of the search process in discovering optimal feature subsets. (2) Limitation 2: Convexity assumptions of the embedding space hinder effective exploration and search process. Prior literature often assumes that the embedding space is convex, with the expectation that gradient-based search will locate optimal solutions. In practice, this assumption rarely holds, which causes the search process to converge to suboptimal points and produce inferior feature subsets. To address these limitations, we propose CAPS , a new framework that performs C ontinuous optimization for fe A ture selection by integrating P ermutation-invariant embeddings with a policy-guided S earch strategy. More specifically, given a large volume of feature selection records, where each record contains the indices of features within a subset and the corresponding model performance, we first design an encoder-decoder framework to capture the underlying patterns in the feature subset indices. To achieve permutation-invariant embeddings for feature subsets, we leverage a self-attention mechanism that symmetrically computes attention scores across all input feature indices. This design guarantees that any permutation of the input features produces identical embeddings, ensuring that the learned representations of feature subset indices are both robust and reliable. Considering the high computational complexity O(N^{2}) of pairwise attention calculation, we introduce a set of inducing points to alleviate the computational burden and accelerate processing. These inducing points act as intermediate representations, facilitating more efficient attention computations by reducing the need for full pairwise attention. This approach achieves a complexity of O(NM) , where M denotes the number of inducing points, which is substantially smaller than N . After the encoder-decoder model converges, we deploy a policy-based reinforcement learning (RL) agent to explore the learned embedding space and identify the optimal feature subset. Based on model performance, we first select the top-K feature subsets as search seeds and input them into the well-trained encoder to obtain their corresponding embeddings. Next, we employ an RL agent to learn how to optimize these embeddings by maximizing the downstream task performance and minimizing the length of the feature subset. Throughout this process, the exploratory nature of the RL agent helps overcome the challenges of the non-convex embedding space, enabling the search for improved embeddings. To summarize, our contributions are three-fold: • Problem: We investigate a new framework for automated feature selection from the generative lens. Remarkably, we propose to solve this problem by integrating permutation-invariant embedding and policy-guided search. • Algorithm: We design a novel encoder-decoder architecture to embed feature selection knowledge into a permutation-invariant space, removing permutation bias. We also employ a policy-based RL agent to explore the space for optimal solutions, overcoming strong convexity assumptions. • Evaluation: We conduct extensive experiments on 14 real-world datasets to validate the effectiveness of our approach.

Read the complete text