A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data.

Menze BH, Kelm BM, Masuch R, Himmelreich U, Bachert P, Petrich W, Hamprecht FA. · 2009

Read A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data. by Menze BH, Kelm BM, Masuch R, Himmelreich U, Bachert P, Petrich W, Hamprecht FA. (2009) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc2724423.txt.

Introduction

Menze and co-authors tested in 2009 a two-step method to classify spectral data, patterns of light or chemical signals with thousands of features. Standard regularized methods handle such data well but cannot drop useless features outright. Random forests, an ensemble of decision trees, can rank feature importance through its Gini score, but its splits fit spectral shapes poorly.

They used Gini importance to select features first, then classified the reduced set with regularized methods. On data sets from medical diagnosis, food science, and other fields, that pairing matched or beat filtering by standard statistical tests. It also beat random forests alone and regularized methods on the full feature set. Feature choice first, then a linear classifier, gave the best of both.

Catalog record

CREATORMenze BH, Kelm BM, Masuch R, Himmelreich U, Bachert P, Petrich W, Hamprecht FA.
YEAR2009
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC2724423

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data By Bjoern H Menze, B Michael Kelm, Ralf Masuch, Uwe Himmelreich, Peter Bachert, Wolfgang Petrich, Fred A Hamprecht Published in: BMC Bioinformatics DOI: 10.1186/1471-2105-10-213 ABSTRACT Abstract Background Regularized regression methods such as principal component or partial least squares regression perform well in learning tasks on high dimensional spectral data, but cannot explicitly eliminate irrelevant features. The random forest classifier with its associated Gini feature importance, on the other hand, allows for an explicit feature elimination, but may not be optimally adapted to spectral data due to the topology of its constituent classification trees which are based on orthogonal splits in feature space. Results We propose to combine the best of both approaches, and evaluated the joint use of a feature selection based on a recursive feature elimination using the Gini importance of random forests' together with regularized classification methods on spectral data sets from medical diagnostics, chemotaxonomy, biomedical analytics, food science, and synthetically modified spectral data. Here, a feature selection using the Gini feature importance with a regularized classification by discriminant partial least squares regression performed as well as or better than a filtering according to different univariate statistical tests, or using regression coefficients in a backward feature elimination. It outperformed the direct application of the random forest classifier, or the direct application of the regularized classifiers on the full set of features. Conclusion The Gini importance of the random forest provided superior means for measuring feature relevance on spectral data, but – on an optimal subset of features – the regularized classifiers might be preferable over the random forest classifier, in spite of their limitation to model linear dependencies only. A feature selection based on Gini importance, however, may precede a regularized linear classification to identify this optimal subset of features, and to earn a double benefit of both dimensionality reduction and the elimination of noise from the classification task. BACKGROUND The high dimensionality of the feature space is a characteristic of learning problems involving spectral data. In many applications with a biological or biomedical background addressed by, for example, nuclear magnetic resonance or infrared spectroscopy, also the number of available samples N is lower than the number of features in the spectral vector P. The intrinsic dimensionality Pintr of spectral data, however, is often much lower than the nominal dimensionality P – sometimes even below N. Dimension reduction and feature selection in the classification of spectral data Most methods popular in chemometrics exploit this relation Pintr < P and aim at regularizing the learning problem by implicitly restricting its free dimensionality to Pintr. (Here, and in the following we will adhere to the algorithmic classification of feature selection approaches from [1], referring to regularization approaches which explicitly calculate a subset of input features – in a preprocessing, for example – as explicit feature selection methods, and to approaches performing a feature selection or dimension reduction without calculating these subsets as implicit feature selection methods.) Popular methods in chemometrics, such as principal component regression (PCR) or partial least squares regression (PLS) directly seek for solutions in a space spanned by ~Pintr principal components (PCR) – assumed to approximate the intrinsic subspace of the learning problem – or by biasing projections of least squares solutions towards this subspace [2,3], down-weighting irrelevant features in a constrained regression (PLS). It is observed, however, that although both PCR and PLS are capable learning methods on spectral data – used for example for in [4] – they still have a need to eliminate useless predictors [5,6]. Thus, often an additional explicit feature selection is pursued in a preceding step to eliminate spectral regions which do not provide any relevant signal at all, showing resonances or absorption bands that can clearly be linked to artefacts, or features which are unrelated to the learning task. Discarding irrelevant feature dimensions, though, raises the question of how to choose such an appropriate subset of features [6-8]. Different univariate and multivariate importance measures can be used to rank features and to select them accordingly [1]. Univariate tests marginalize over all but one feature and rank them in accordance to their discriminative power [9,10]. In contrast, multivariate approaches consider several or all features simultaneously, evaluating the joint distribution of some or all features and estimating their relevance to the overall learning task. Multivariate tests are often used in wrapper schemes in combination with a subsequent classifier (e.g. a global optimization of feature subset and classifier coefficients [11]), or by statistical tests on the outcome of a learning algorithm (e.g. an iterative regression with test for robustness [12,13]). While univariate approaches are sometimes deemed too simplistic, the other group of multivariate feature selection methods often comes at unacceptably high computational costs. Gini feature importance A feature selection based on the random forest classifier [14] has been found to provide multivariate feature importance scores which are relatively cheap to obtain, and which have been successfully applied to high dimensional data, arising from microarrays [15-20], time series [21], even on spectra [22,23]. Random forest is an ensemble learner based on randomized decision trees (see [24] for a review of random forests in chemometrics, [14] for the original publication, and [25-28] for methodological aspects), and provides different feature important measures. One measure is motivated from statistical permutation tests, the other is derived from the training of the random forest classifier. Both measures have been found to correlate reasonably well [28]. While the majority of the prior studies focused on the first, we will focus on the second in the following. As a classifier, random forest performs an implicit feature selection, using a small subset of "strong variables" for the classification only [27], leading to its superior performance on high dimensional data. The outcome of this implicit feature selection of the random forest can be visualized by the "Gini importance" [14], and can be used as a general indicator of feature relevance. This feature importance score provides a relative ranking of the spectral features, and is – technically – a by-product in the training of the random forest classifier: At each node τ within the binary trees T of the random forest, the optimal split is sought using the Gini impurity i(τ) – a computationally efficient approximation to the entropy – measuring how well a potential split is separating the samples of the two classes in this particular node. With being the fraction of the nk samples from class k = {0,1} out of the total of n samples at node τ, the Gini impurity i(τ) is calculated as Its decrease Δi that results from splitting and sending the samples to two sub-nodes τl and τr (with respective sample fractions and ) by a threshold tθ on variable θ is defined as In an exhaustive search over all variables θ available at the node (a property of the random forest is to restrict this search to a random subset of the available features [14]), and over all possible thresholds tθ, the pair {θ, tθ} leading to a maximal Δi is determined. The decrease in Gini impurity resulting from this optimal split Δiθ (τ,

Read the complete text