Classifier Chain Networks for Multi-Label Classification

Daniel J.W. Touw, Michel van de Velden · 2024

Read Classifier Chain Networks for Multi-Label Classification by Daniel J.W. Touw, Michel van de Velden (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02638.txt.

Description

Abstract The classifier chain is a widely used method for analyzing multi-labeled data sets. In this study, we introduce a generalization of the classifier chain: the classifier chain network . The classifier chain network enables joint estimation of model parameters, and allows to account for the influence of earlier label predictions on subsequent classifiers in the chain.

Catalog record

CREATORDaniel J.W. Touw, Michel van de Velden
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Classifier Chain Networks for Multi-Label Classification Daniel J.W. Touw, Michel van de Velden Abstract The classifier chain is a widely used method for analyzing multi-labeled data sets. In this study, we introduce a generalization of the classifier chain: the classifier chain network . The classifier chain network enables joint estimation of model parameters, and allows to account for the influence of earlier label predictions on subsequent classifiers in the chain. Through simulations, we evaluate the classifier chain network’s performance against multiple benchmark methods, demonstrating competitive results even in scenarios that deviate from its modeling assumptions. Furthermore, we propose a new measure for detecting conditional dependencies between labels and illustrate the classifier chain network’s effectiveness using an empirical data set. Keywords: multi-label classification, classifier chain, simultaneous parameter estimation, conditional dependency 1 Introduction A multi-label classifier models the association of an observation with multiple labels. In contrast to binary and multi-class classification, where each observation in the data is assigned to a single class, an observation in a multi-label classification task can have multiple labels. This type of problem arises in different fields, such as object detection in images, text analysis, bioinformatics, and recommendation systems ( Tsoumakas et al.,, 2010 ) . Consequently, numerous methods have been developed to handle multi-labeled outcomes. In contrast to existing methods, which often focus on modeling each outcome variable separately, our proposed method jointly models all labels to capture dependencies between them. In this study, we also refer to these dependencies between labels as label interdependencies . A frequently used method for a classification task with multi-labeled outcomes is to decompose the task into separate independent binary classifications ( Boutell et al.,, 2004 ; Luaces et al.,, 2012 , e.g.,) . This approach is typically referred to as binary relevance . A limitation of binary relevance is the fact that it does not exploit potential correlations between the different labels ( Godbole and Sarawagi,, 2004 ; Zhang and Zhou,, 2014 ) . Empirical evidence supports the notion that models explicitly accounting for possible dependencies between labels may outperform binary relevance ( Ji et al.,, 2008 ; Loza Mencía and Fürnkranz,, 2008 ; Liu and Chen,, 2015 , e.g.,) . A number of methods have been developed that simultaneously model a multi-labeled outcome and take into account label interdependencies. Early examples are AdaBoost.MH ( Schapire and Singer,, 1998 ) , multi-label decision trees ( Clare and King,, 2001 ) , multi-label k -nearest neighbors ( Zhang and Zhou,, 2005 ) , random k -labelsets ( Tsoumakas et al.,, 2011 ) , instance-based learning combined by logistic regression ( Cheng and Hüllermeier,, 2009 ) , and classifier chains ( Read et al.,, 2009 ; Read et al.,, 2011 ) . In this study, we focus on the latter because it has shown favorable results in the existing literature ( Moyano et al.,, 2018 ; Rivolli et al.,, 2020 , e.g.,) . The classifier chain is closely related to binary relevance, as both methods use binary classifiers to predict the occurrence of each of the L labels. However, it extends binary relevance by accommodating label interdependencies ( Zhang et al.,, 2018 ) . This is achieved by assuming a sequential order in the occurrence of the labels and chaining the classifiers accordingly, each using the preceding labels as additional explanatory variables. As a result, each label is modeled using more information than its predecessors. A notable property of the classifier chain is the reliance on the ordering of the labels, which may not always be known in practical applications. Read et al., (2009) ; Read et al., (2011) address this challenge by introducing an ensemble of classifier chains, combining multiple chains, each with randomly ordered sequences of labels. Classifier chains have shown to be a strong performer for multi-label classification tasks. For example, Moyano et al., (2018) and Bogatinovski et al., (2022) demonstrated that the ensemble classifier chain is among the top performers across various evaluation metrics. This observation has led to a number of studies seeking to expand and improve the method. Variations are, among others, probabilistic classifier chains ( Dembczyński et al., 2010a, ) , recurrent classifier chains ( Nam et al.,, 2017 ) , and nested stacking ( Senge et al.,, 2019 ) . Furthermore, Weng et al., (2020) show that label-specific features can improve out-of-sample performance and Jun et al., (2019) propose a method to forgo ensembles of classifier chains and order the labels based on their conditional entropy. In this paper, we propose a generalization of the classifier chain that conceptualizes the model as a single network. Unlike traditional classifier chains, which fit classifiers sequentially and are “blind” to the impact of their predictions on subsequent classifiers, our approach models all outcomes simultaneously. Consequently, our model is able to take into account the impact of the prediction for the first label on the prediction for the last label. We call this generalization the classifier chain network . Our approach aligns with previous studies, such as Nam et al., (2017) , which treat multi-label classification tasks as viable targets for neural network architectures. However, in contrast to deep learning methodologies, we use modeling assumptions to constrain the number of parameters, ensuring an interpretable model applicable to small to medium-sized data sets. As part of the proposed generalization, we incorporate a regularization step into the loss function used to estimate the model parameters. This adjustment, inspired by Van den Burg and Groenen, (2016) , aims to mitigate an overemphasis on observations with many misclassified labels. To gain insights into the performance of the classifier chain network relative to other multi-label classification methods, we conduct a comprehensive simulation study. These simulations provide a controlled environment where the ground truth is known, allowing us to identify specific scenarios and conditions that highlight the strengths and limitations of the classifiers under consideration. The results show that the classifier chain network generally outperforms benchmark methods across multiple performance metrics and simulation designs. Notably, it achieves excellent results in terms of the negative log-likelihood, reflecting a desirable balance of higher confidence in correct predictions and appropriately lower confidence in incorrect ones. Exploring an issue related to multi-label classification tasks, we examine in which scenarios methods that explicitly account for label interdependencies offer benefits over binary relevance. Prior research has noted cases where binary relevance demonstrates competitive performance ( Luaces et al.,, 2012 ; Douibi et al.,, 2019 , e.g.) . Additionally, it has been established that for methods explicitly modeling label interdependencies to be advantageous, there must be conditional dependency between the labels ( Dembczyński et al., 2010b, ) . We propose a new measure for detecting conditional dependencies and compare it to existing approaches. The remainder of this paper is structured as follows. In Section 2 , we present the classifier chain network and its estimation procedure. To compare our method with alternative multi-label classification methods, a simulation study is performed in Section 3 and we propose a new method to detect conditional dependencies between labels in Section 4 . In Section 5 , we present an application of the classifier chain network to empirical data, and Section 6 concludes the paper. 2 Classifier Chain Network This section provides an

Read the complete text