Dynamic Feature Selection based on Rule-based Learning for Explainable Classification with Uncertainty Quantification

Javier Fumanal-Idocin, Raquel Fernandez-Peralta, Javier Andreu-Perez · 2025

Read Dynamic Feature Selection based on Rule-based Learning for Explainable Classification with Uncertainty Quantification by Javier Fumanal-Idocin, Raquel Fernandez-Peralta, Javier Andreu-Perez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02566.txt.

Introduction

Javier Fumanal-Idocin, Raquel Fernandez-Peralta, and Javier Andreu-Perez tackle feature choice under a budget in this 2025 paper. In many tasks, each added feature costs time or money, so a model should ask for data step by step. Past methods for this need models built for that flow. They also give weak estimates of how unsure a result is.

The head text gives the work a plainer title about model-free dynamic feature choice. The authors show that step-by-step choice adds new sources of doubt. They then offer a method that works with trained models already in use, even rule-based ones. Tests on table and image data match top methods while keeping results more open to check.

Catalog record

CREATORJavier Fumanal-Idocin, Raquel Fernandez-Peralta, Javier Andreu-Perez
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02566

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamic Feature Selection based on Rule-based Learning for Explainable Classification with Uncertainty Quantification Javier Fumanal-Idocin, Raquel Fernandez-Peralta, Javier Andreu-Perez Abstract Dynamic feature selection (DFS) offers a compelling alternative to traditional, static feature selection by adapting the selected features to each individual sample. Unlike classical methods that apply a uniform feature set, DFS customizes feature selection per sample, providing insight into the decision-making process for each case. DFS is especially significant in settings where decision transparency is key, i.e., clinical decisions; however, existing methods use opaque models, which hinder their applicability in real-life scenarios. This paper introduces a novel approach leveraging a rule-based system as a base classifier for the DFS process, which enhances decision interpretability compared to neural estimators. We also show how this method provides a quantitative measure of uncertainty for each feature query and can make the feature selection process computationally lighter by constraining the feature search space. We also discuss when greedy selection of conditional mutual information is equivalent to selecting features that minimize the difference with respect to the global model predictions. Finally, we demonstrate the competitive performance of our rule-based DFS approach against established and state-of-the-art greedy and RL methods, which are mostly considered opaque, compared to our explainable rule-based system. Introduction Many real-world decision-making scenarios involve prohibitively high-dimensional information spaces, where acquiring all possible data is unfeasible due to cost or time constraints ( Erion et al. 2022 ) . Consider, for instance, a doctor diagnosing a patient. The doctor must select only the right clinical trials to conduct, as they are expensive in terms of both money and time, which are usually scarce resources. Only a small subset will be truly relevant in accurately diagnosing the patient’s specific condition, and this subset will be different for each patient. The most intuitive and effective approach in such situations is to adapt the information acquisition process to the subject’s condition. This methodology, where features are acquired dynamically based on their evolving relevance, is commonly referred to as Dynamic Feature Selection (DFS) ( Melville et al. 2004 ) . The existing machine learning literature on DFS primarily explores two main approaches: reinforcement learning (RL) ( Liu et al. 2021 ; Janisch, Pevnỳ, and Lisỳ 2019 ) and greedy optimization ( Chen et al. 2015 ) . RL-based approaches offer the potential to discover optimal policies for feature acquisition, but they come with the inherent challenges of RL training, including significant computational demands and potential convergence issues ( Erion et al. 2022 ) . Greedy optimization methods, on the other hand, maximize a measure that relates the known inputs with the response variable ( Chattopadhyay et al. 2023 ) . Usually, this measure is the conditional mutual information (CMI) between input variables and the response variable, which has some desirable qualities in terms of uncertainty and prediction power ( Covert et al. 2023 ; Gadgil, Covert, and Lee 2023 ) . Significant research within CMI-based DFS methods focuses on the task of accurately estimating CMI itself, which often requires strong prior assumptions about the data distribution. It is possible to use a generative model to learn the feature distribution, which then makes the CMI estimation straightforward ( Ma et al. 2018 ) , though these can be computationally expensive to train and lead to slow inference. More recent research has also proposed gradient-based optimization with a neural regressor for CMI estimation, which does not need generative models, using a value network with a loss function that recovers CMI at optimality ( Gadgil, Covert, and Lee 2023 ) . Other strategies bypass direct CMI computation, focusing instead on identifying the feature with maximum CMI without explicitly calculating its value ( Covert et al. 2023 ) . A good property of these CMI-based estimation models is that maximizing CMI during the selection process naturally leads to uncertainty minimization ( Chen et al. 2018 ) . However, these methods have been tied to neural-based classifiers, which is not good for explainability. For example, suppose we want to analyze the impact of each feature in the final classification. In that case, we still need to use a post hoc method, such as LIME or SHAP, even when an intuitive policy has controlled the feature selection process. Recent works have focused on using interpretable-by-design models, which makes decisions easier to understand by a human user ( Rudin 2019 ; Molnar 2020 ) . Families of such methods include rule-based and tree-based models, where the output is the direct consequent of one or various logical predicates ( Marques-Silva 2023 ) , linear and logistic regressions, whose weights clearly reveal the relationship between feature and prediction ( Letzgus et al. 2022 ) , and nearest neighbour models, like prototype models, where the output is explained by their similarity to other samples where ground truth was available ( Nauta, Van Bree, and Seifert 2021 ; Fernández et al. 2022 ) . Besides, the notion of uncertainty in previous works is constrained to the relations between variables, but they do not consider how well the model approximates those variables. Our focus in this paper is to create a robust DFS algorithm that is intuitive and explainable to the user and has sound uncertainty management. This allows the user to have a transparent feature acquisition process, where the information gained in each feature query is clear, and the final decision is also interpretable. To do that, we focus on the following issues regarding DFS: • We propose to use a global model that is trained on the whole set of features as an oracle model. This will allow us to model uncertainty without needing more data than other approaches. • We discuss the advantages of using interpretable-by-design rule-based classifiers for interpretability and uncertainty estimation. For that, we also study both fuzzy and classical logic-based rules. • We adapt the aleatoric uncertainty estimation error of the existing CMI-based DFS approaches to rule-based models, and we propose a new way of estimating the epistemic uncertainty, which has not been considered by those methods. Background Dynamic Feature Selection: Problem Formulation Let \mathbf{x}=(x_{1},\dots,x_{M}) be an input feature vector and y\in\{1,\dots,C\} a corresponding target label in a supervised learning paradigm. The complete set of N input samples is denoted as X=\{\mathbf{x}^{1},\dots,\mathbf{x}^{N}\} . A partially observed sample, where only features indexed by S have been observed, is denoted as \mathbf{x}_{S}=\{x_{i}:i\in S\} . The set of unobserved features is denoted as \bar{S}=\{1,\dots,M\}\setminus S . The objective of DFS is to design a feature selection policy that aims to correctly classify each sample by selecting the most informative features. This policy adapts the selection process based on the specific characteristics of each instance, improving both classification accuracy and interpretability. Conditional Mutual Information for Dynamic Feature Selection Conditional Mutual Information (CMI) provides a principled measure for evaluating feature relevance in dynamic feature selection. Given three random variables X , Y , and Z , the CMI quantifies the information shared between X and Y when Z is known. We present two equivalent formulations: I(X;Y|Z)=H(X|Z)-H(X|Y,Z), (1) where H(\cdot|\cdot) denotes the conditional entropy. This formulation measures the reduction in uncertainty about X when Y is observed, beyond what is already known from Z . The CMI can also

Read the complete text