Making Reliable and Flexible Decisions in Long-tailed Classification

Bolian Li [email protected] · 2025

Read Making Reliable and Flexible Decisions in Long-tailed Classification by Bolian Li [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14090.txt.

Introduction

Bolian Li and Ruqi Zhang tackle, in a 2025 paper, classification where a few classes dominate and mistakes are not priced equally. In long-tailed problems, calling a sick patient healthy is far worse than the reverse, yet most methods chase plain accuracy.

Their framework, RF-DLC, folds the cost of each error type directly into learning through Bayesian decision theory, using an integrated gain that respects both the skewed data and the decision rule. A variational method makes the training efficient, and any cost table can be plugged in. A new metric, the false head rate, measures tail risk. Tests covered large image tasks and uncertainty checks.

Catalog record

CREATORBolian Li [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14090

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Making Reliable and Flexible Decisions in Long-tailed Classification Bolian Li [email protected] Abstract Long-tailed classification is challenging due to its heavy imbalance in class probabilities. While existing methods often focus on overall accuracy or accuracy for tail classes, they overlook a critical aspect: certain types of errors can carry greater risks than others in real-world long-tailed problems. For example, misclassifying patients (a tail class) as healthy individuals (a head class) entails far more serious consequences than the reverse scenario. To address this critical issue, we introduce Making R eliable and F lexible D ecisions in L ong-tailed C lassification (RF-DLC), a novel framework aimed at reliable predictions in long-tailed problems. Leveraging Bayesian Decision Theory, we introduce an integrated gain to seamlessly combine long-tailed data distributions and the decision-making procedure. We further propose an efficient variational optimization strategy for the decision risk objective. Our method adapts readily to diverse utility matrices, which can be designed for specific tasks, ensuring its flexibility for different problem settings. In empirical evaluation, we design a new metric, False Head Rate, to quantify tail-sensitivity risk, along with comprehensive experiments on multiple real-world tasks, including large-scale image classification and uncertainty quantification, to demonstrate the reliability and flexibility of our method. 1 1 1 https://github.com/lblaoke/RF-DLC . Reviewed on OpenReview: https://openreview.net/forum?id=hMO8sT9qaD 1 Introduction Real-world categorical data is often long-tailed distributed, where the data distributions are biased towards a few “head” classes and the “tail” classes have much fewer samples ( Reed, 2001 ; Lin et al., 2014 ; Van Horn & Perona, 2017 ; Krishna et al., 2017 ; Liu et al., 2019b ; Wang et al., 2020 ; Li et al., 2022 ) . The long-tailed problem primarily stems from the inherent biases in the data collection process, which are challenging to avoid. Models trained on long-tailed data using conventional methods often exhibit notable performance drops compared to those trained on balanced datasets ( Wang et al., 2022 ) . Existing approaches to long-tailed classification, such as re-weighting loss ( Lin et al., 2017 ; Cao et al., 2019 ; Cui et al., 2019 ; Wu et al., 2020 ) , logit adjustment ( Menon et al., 2020 ; Ren et al., 2020 ; Hong et al., 2021 ) , and knowledge transfer ( Liu et al., 2019b ; Xiang et al., 2020 ) , usually focus on improving overall evaluation metrics or the metrics for tail classes. However, in real-world long-tailed data, the misprediction penalty between different classes often depends on their semantic meaning ( He et al., 2024 ) , and the ultimate objective is making optimal decisions rather than optimizing certain metrics. The probability of misclassifying tailed samples as head samples is very high in existing methods. For example, in disease detection where healthy individuals belong to head classes and patients belong to tail classes, the risk of classifying patients as healthy individuals is significantly larger than the reverse scenario ( Yang et al., 2022 ) . In autonomous driving, mispredicting tail classes like pedestrians or cyclists as head classes like vehicles will significantly increase the risk of accidents and injuries ( Carranza-García et al., 2021 ) . To enable reliable long-tailed classification and simultaneously integrate decision-making, we propose Making R eliable and F lexible D ecisions in L ong-tailed C lassification (RF-DLC), a general learning framework aimed at reliable predictions 2 2 2 The words “prediction” and “decision” are used interchangeably throughout this paper. on diverse realistic long-tailed problems. Specifically, we introduce the integrated gain from Bayesian Decision Theory ( Robert et al., 2007 ; Berger, 2013 ) , an important branch of cost-sensitive learning, which allows us to seamlessly incorporate decision risk and long-tailed data distributions in a single objective. While cost-sensitive learning has been mainly applied to standard classification ( Elkan, 2001 ; He & Garcia, 2009 ; Thai-Nghe et al., 2010 ; Chung et al., 2016 ; Shu et al., 2019 ) , its application to long-tailed classification is limited to the case of avoiding misclassifying tail samples as head. We also propose a variational optimization strategy to efficiently maximize the integrated gain w.r.t. model parameters. With “utility matrix” as a part of the objective, our method is flexible to many real-world tasks with different types of risks. In our empirical evaluations, we design a new metric, False Head Rate (FHR), to quantity the mispredictions from tail classes to head classes–a common source of high risk in long-tailed classification ( Sengupta et al., 2016 ; Rahman et al., 2021 ; Yang et al., 2022 ) . The main contributions of this paper are summarized as follows: • RF-DLC is the first to consider decision-making in long-tailed classification. Built upon Bayesian Decision Theory, RF-DLC enables optimal decision-making on long-tailed data. • RF-DLC introduces a new objective called integrated gain, an efficient variational optimization strategy, and several utility matrices tailored for different long-tailed scenarios. All of these are directly derived from Bayesian Decision Theory, ensuring that they are integral to the model’s function. • RF-DLC is flexible and can be adapted to various tasks with diverse metrics. Users can adopt RF-DLC to many specific fields by re-designing utility matrices. • We conduct comprehensive experiments to demonstrate that RF-DLC significantly improves decision-making while maintaining or improving traditional metrics such as accuracy and calibration. 2 Related Works Long-tailed Classification. Previous methods mainly tackle long-tailed classification from the following aspects: i) adjusting data distributions to obtain balanced datasets, including over-sampling ( Han et al., 2005 ) , under-sampling ( Liu et al., 2008 ) , and data augmentation ( Chu et al., 2020 ; Kim et al., 2020 ; Liu et al., 2020 ) 3 3 3 These methods fall behind the SOTA of long-tailed classification significantly, and thus are not compared in the experiments. ; ii) re-balancing the importance of different classes in loss functions, including re-weighting loss ( Lin et al., 2017 ; Mahajan et al., 2018 ; Cao et al., 2019 ; Cui et al., 2019 ; Menon et al., 2020 ; Wu et al., 2020 ) and logit adjustment ( Menon et al., 2020 ; Ren et al., 2020 ; Hong et al., 2021 ) ; iii) applying heterogeneous model architectures to handle head and tail samples in different ways, including OLTR ( Liu et al., 2019b ) , LFME ( Xiang et al., 2020 ) , RIDE ( Wang et al., 2020 ) , TLC ( Li et al., 2022 ) , and SRepr ( Nam et al., 2023 ) . Other attempts explore broader long-tailed classifications, including non-uniform testing distributions ( Zhang et al., 2022 ) , outlier samples ( Wang et al., 2022 ; Bai et al., 2022 ) , and partial-labeled datasets ( Hong et al., 2022 ) . Our method is significantly different from existing methods, targeting at decision-making and asymmetric misprediction risks, which are important problems in realistic long-tailed data. Bayesian Decision Theory and Cost-sensitive Learning. Robert et al. (2007) ; Berger (2013) have comprehensively introduced Bayesian Decision Theory, which enables optimal decisions under diverse problem settings (via the choices of utility functions). For its adaptation to deep neural networks, multiple ways have been explored: i) loss-calibrated variational inference, including Loss-calibrated EM ( Lacoste-Julien et al., 2011

Read the complete text