Distributionally Robust Feature Selection
Maitreyi Swaroop, Tamar Krishnamurti, Bryan Wilder · 2025
Read Distributionally Robust Feature Selection by Maitreyi Swaroop, Tamar Krishnamurti, Bryan Wilder (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21113.txt.
Introduction
Maitreyi Swaroop leads two co-authors on distributionally robust feature selection, choosing a small set of features to observe so models trained on them perform well across many subpopulations at once. Features can be costly, each survey question or sensor adds burden, so the chosen few must serve everyone.
Their method relaxes the discrete choice into a continuous problem through a noising mechanism, avoiding any need to backpropagate through model training. Optimization targets worst case quality across groups rather than average quality alone. Robust selection suits health surveys and sensing deployments where one instrument must serve diverse communities.
Catalog record
| CREATOR | Maitreyi Swaroop, Tamar Krishnamurti, Bryan Wilder |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21113
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distributionally Robust Feature Selection Maitreyi Swaroop, Tamar Krishnamurti, Bryan Wilder Abstract We study the problem of selecting limited features to observe such that models trained on them can perform well simultaneously across multiple subpopulations. This problem has applications in settings where collecting each feature is costly, e.g. requiring adding survey questions or physical sensors, and we must be able to use the selected features to create high-quality downstream models for different populations. Our method frames the problem as a continuous relaxation of traditional variable selection using a noising mechanism, without requiring backpropagation through model training processes. By optimizing over the variance of a Bayes-optimal predictor, we develop a model-agnostic framework that balances overall performance of downstream prediction across populations. We validate our approach through experiments on both synthetic datasets and real-world data. 1 1 1 Code for implementing our method is available here (linked) . 1 Introduction Many real-world applications impose significant constraints on data collection for machine learning models. These constraints often stem from factors such as limited budgets, privacy concerns, or the operational costs associated with acquiring each data point. For instance, in healthcare, a hospital system aiming to implement a predictive screener across diverse patient populations must often contend with limitations on the number of questions permissible due to patient burden, time constraints, and regulatory considerations. In such scenarios, the ability to identify a minimal yet highly informative set of features— feature selection —becomes paramount. Oftentimes, we might collect pilot data on a large number of features in order to make the operational decision of which to collect in deployment. For example, medical screeners are often developed with a larger number of questions that are then cut down to a smaller number for general-population usage; e.g., the PHQ ( Spitzer et al. 1999 ) is a 26-item mental health survey that is typically reduced to standard 9 or 2-question versions in practice ( Kroenke et al. 2001 ; Kroenke et al. 2003 ) . The challenge intensifies when the selected features must ensure reliable model performance not just on average, but across varied and potentially shifting underlying data distributions. This necessitates a distributionally robust approach to feature selection, ensuring that no particular subpopulation is inadvertently neglected by models trained on the selected features. Furthermore, in many practical settings, the specific downstream inference model that will ultimately use the collected data may not be known at the time of feature selection, or multiple different models might be employed. Therefore, an ideal feature selection methodology should be model-agnostic , providing a core set of features that are broadly useful without being tied to the idiosyncrasies of a particular predictive algorithm. This paper addresses the problem of selecting a limited subset of features from a larger pool, using historical data, to facilitate accurate and robust predictions across diverse populations, all while adhering to a collection budget. We aim to identify features that are robust to distributional shifts, ensuring that the model performs well simultaneously across a variety of specified subpopulations (e.g., institutional settings, clinics, or demographic groups). To our knowledge, this problem has not been previously studied in the literature: feature selection methods focus on a single distribution of interest, while distributionally robust optimization (DRO) methods attempt to find a single model that performs well on all populations, instead of selecting a limited number of features that can then be used to train high-performing models for each downstream population. The intersection of the two problems complicates both. On the one hand, we require an approach to feature selection that is model-agnostic and naturally extends across multiple populations. On the other hand, standard DRO methods do not apply because we have a discrete optimization problem of selecting features instead of directly optimizing over a single model. Our approach formulates this task by introducing a continuous relaxation of the discrete feature selection problem which injects synthetic noise into the observation of each covariate. We then develop an analytical simplification of the relaxation which eventually allows us to solve it with standard stochastic gradient descent-style methods. Our main contributions are as follows: 1. We propose a novel, model-agnostic method for distributionally robust feature selection. This method identifies a subset of features that minimizes the maximum expected loss (or error) across potential data distributions, without requiring assumptions about the specific architecture or differentiability of the downstream predictive model. 2. We reframe the combinatorial problem of feature selection into a continuous optimization framework. This is achieved by introducing parameters that govern a noise injection process, effectively creating a differentiable measure of each feature’s utility and permitting tractable, gradient-based optimization for selection. 3. We demonstrate the efficacy of our approach through experiments on both synthetic and real-world datasets, comparing its performance against naive selection strategies and existing distributionally robust optimization (DRO) baselines adapted for feature selection. 1.1 Related work Our work lies in the intersection of distributionally robust optimization and feature selection. While there is a rich body of work that focuses on each topic individually, to our knowledge no previous work studies the intersection – selecting a set of features that will allow a high-performing model to be trained for each subpopulation, as opposed to training a single model that works well everywhere. Feature selection: Classical feature selection methods primarily fall into two categories: combinatorial optimization approaches ( Guyon and Elisseeff 2003 ; Kohavi and John 1997 ) , and embedded methods ( Breiman 2001 ; Tibshirani 1996 ) . In the combinatorial optimization vein, a long line of work develops greedy strategies or forward/backward selection heuristics for variable selection, historically focused on linear models. Even for linear models, selecting the best set of at most k features is known to be NP-hard ( Natarajan 1995 ) . Das and Kempe 2011 , provided theoretical approximation guarantees for greedy selection in linear models via a connection to submodularity. Khanna et al. 2017 provide faster algorithms for the same problem. However, there is no direct path to extend these methods beyond linear models, or to incorporate robustness across multiple distributions as a goal. Lasso regression ( Tibshirani 1996 ) is a classical method frequently used for feature selection, which leverages an \ell_{1} penalty to induce sparsity. However, the Lasso formulation does not account for robustness to distributional shifts and again bakes in an assumption of linearity. Zhao and Yu 2006 highlight potential pitfalls of Lasso-based feature selection, showing that it may not consistently select the correct variables under certain correlation structures. Recent advances in embedded methods have explored heterogeneous feature selection, though these address different problem settings from our work. Yang et al. 2022 propose locally sparse neural networks (LSPIN) that learn sample-specific feature subsets, allowing different features for different samples. Similarly, Svirsky and Lindenbaum 2024 introduce interpretable deep clustering (IDC) that performs cluster-level feature selection. While these methods advance feature selection for heterogeneous populations, they fundamentally differ from our approach: our method selects a single global feature subset that must perform well universally across all known population groups, a constraint motivated by practical deployment scenarios where systems require a fixed set of features for all users. Previous work has also proposed the use of random noise injection for feature selection in single population settings (