Discriminative Subspace Emersion from learning feature relevances across different populations

Marco Canducci, Lida Abdi, Alessandro Prete, Roland J. Veen, Michael Biehl, Wiebke Arlt, Peter Tino · 2025

Read Discriminative Subspace Emersion from learning feature relevances across different populations by Marco Canducci, Lida Abdi, Alessandro Prete, Roland J. Veen, Michael Biehl, Wiebke Arlt, Peter Tino (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00176.txt.

Introduction

Marco Canducci and six co-authors ask which traits split two groups in twenty twenty-five. A class task is hard when the training set is small, the traits are many, and the class scores overlap a lot. When two split groups share the same task, the aim of the work shifts. What counts most is which traits best split how the task plays out in each group.

Their tool, Discriminative Subspace Emersion, grows subspace learning into a broad trait rank plan. It finds the traits that count most to tell how the two groups make their class calls. The plan builds on clear learners with near straight cut lines. Careful tuning and trait rank scores do most of the hard work. The tool lets you set side by side how a choice is made in each group, and not just how right that choice is.

Catalog record

CREATORMarco Canducci, Lida Abdi, Alessandro Prete, Roland J. Veen, Michael Biehl, Wiebke Arlt, Peter Tino
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00176

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Discriminative Subspace Emersion from learning feature relevances across different populations Marco Canducci, Lida Abdi, Alessandro Prete, Roland J. Veen, Michael Biehl, Wiebke Arlt, Peter Tino Abstract In a given classification task, the accuracy of the learner is often hampered by finiteness of the training set, high-dimensionality of the feature space and severe overlap between classes. In the context of interpretable learners, with (piecewise) linear separation boundaries, these issues can be mitigated by careful construction of optimization procedures and/or estimation of relevant features for the task. However, when the task is shared across two disjoint populations the main interest is shifted towards estimating a set of features that discriminate the most between the two, when performing classification. We propose a new Discriminative Subspace Emersion (DSE) method to extend subspace learning toward a general relevance learning framework. DSE allows us to identify the most relevant features in distinguishing the classification task across two populations, even in cases of high overlap between classes. The proposed methodology is designed to work with multiple sets of labels and is derived in principle without being tied to a specific choice of base learner. Theoretical and empirical investigations over synthetic and real-world datasets indicate that DSE accurately identifies a common subspace for the classification across different populations. This is shown to be true for a surprisingly high degree of overlap between classes. keywords Subspace Learning, Feature Relevance, Metric Learning, Generalized Matrix Learning Vector Quantization (GMLVQ), Support Vector Machines (SVM) † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction In supervised learning, annotated data is used to extract information about the properties that most distinguish the classes in which data is organized. In this sense, classification generally aims at recovering a boundary that separates the given classes. It is often good practice to assume that not all features are equally relevant in this discrimination, but some display large variability across the different data categories. The goal of finding a subset of relevant features for the classification task is often referred to as subspace learning. Extreme examples of this approach are Support Vector Machine (SVM, Cortes and Vapnik (1995) ) and logistic regression. SVM performs linear separation by finding the optimal hyperplane that separates the classes in the input space. Logistic regression performs linear separation by modelling the probability that a given input belongs to a particular class. A different category of supervised classification algorithms approach the discrimination between the classes via the use of prototypes, encapsulating and compressing the properties of the classes into a small set of typical examples. Examples of prototype-based methodologies are the different implementations of Learning Vector Quantization (LVQ, Kohonen (1995) ). Instead of finding a subset of meaningful features for the classification, metric learning methodologies ( Kulis (2013) and references therein) focus on determining a metric tensor that endows the feature space with a geodesic distance. By pushing similar points (members of the same class) closer and dissimilar points (members of different classes) further apart, this distance acts as a dissimilarity measure. When the constraints of the dissimilarity matrix are relaxed, in particular positive definiteness, the learning process might discard irrelevant features in the classification task, de facto extracting a meaningful subset that defines a subspace of the original space. Although in this case the dissimilarity matrix is not exactly a metric tensor, its reduction to the lower-dimensional subspace is. Thus, slightly abusing notation, in the following we will refer to this matrix as the metric tensor. Generalized Matrix Learning Vector Quantization (GMLVQ, Schneider et al. (2009) ) is a prototype-based classification algorithm that also estimates a dissimilarity metric from the data. GMLVQ aims to achieve piecewise linear separation between classes by adjusting the prototype vectors and the relevance matrix such that data points from different classes are well-separated in the feature space. GMLVQ has successful applications in various domains such as pattern recognition, image classification, and bio-informatics Biehl et al. (2013) ; Veen et al. (2020) . The adaptive metric tensor accounts for the correlation between the features, and it provides information about the structure of the data. The diagonal elements of the metric tensor indicate the importance of features ( relevances ) and off-diagonal elements indicate correlation and dependencies of the features. It implicitly identifies important features through the learned discriminative subspace. Discriminative subspace learning focuses on projecting high-dimensional data into a lower-dimensional space while maintaining class separability. Chen and Kortje (2025) propose a supervised dimension reduction method using linear projection and Kullback-Leibler divergence to optimize feature separability. Vogelstein et al. (2021) extend this approach to big data, leveraging scalable algorithms for large datasets. Yin et al. (2023) employ Riemannian manifold optimization for enhanced class discrimination. Fu et al. (2022) ; Fu et al. (2023) develop and evaluate the Subspace Learning Machine (SLM), incorporating decision trees and nonhomogeneous media analysis for optimized classification. Amiri and Modarres (2025) introduce a subspace aggregating algorithm combining bagging, boosting, and random forests for enhanced classification accuracy. Yan et al. (2006) present a scalable supervised subspace learning algorithm, while Yan et al. (2007) establish a graph embedding framework that generalizes various dimensionality reduction methods. Dwivedi et al. (2021) analyze linear discriminant analysis (LDA, Hastie et al. (2001) ) under f-divergence measures to enhance stability. Fukui et al. (2023) introduce generalized difference subspaces for discriminative feature extraction, expanding on prior work by Fukui and Maki (2015) on difference subspaces and their applications in subspace-based methods. Ren et al. (2024) propose a commonality and individuality-based subspace learning approach, integrating multitask learning principles. However, in some applications, identification of important and relevant features is harmed by high overlap between classes. To complicate this scenario, many classification (or subspace learning) algorithms include a level of stochasticity, given by either random initialization or the chosen optimization procedure. Not to mention that if classes are imbalanced, subsampling needs to be adopted to obtain unbiased classifiers. In order to mitigate the effects of these sources of uncertainty, ensemble approaches to classification Sollich and Krogh (1995) ; Zhou (2012) , also referred to as mixture of experts Masoudnia and Ebrahimpour (2014) , can be used Zahavy et al. (2016) . Evidently, a plethora of Subspace Learning algorithms exists that rely on geometrical or informational theoretical notions for identifying a discriminative subspace for the classification task at hand. But when the same classification task needs to be performed over two distinct populations, the focus must be shifted from learning the individual optimal discriminative subspaces, towards identifying the common subspace where populations differ the most (under the lens of the specific classification task). Note that this is different from addressing a multi-label classification problem (e.g. Tarekegn et al. (2021) ) and the corresponding discriminative subspace estimation ( Bayati et al., 2022 ; Ma et al., 2024 ) . To better highlight the scenario proposed in this work, consider the case where the same classification task (presence or not of a health condition in a cohort) has to be performed over two different populations (e.g. patients with different degrees of Cortisol excess). In the following, different populations will be identified by letters “A” and “B” and different health states by “Condition” and “No condition”.

Read the complete text