Difficulty-aware Balancing Margin Loss for Long-tailed Recognition

Minseok Son, Inyong Koo, Jinyoung Park, Changick Kim · 2024

Read Difficulty-aware Balancing Margin Loss for Long-tailed Recognition by Minseok Son, Inyong Koo, Jinyoung Park, Changick Kim (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15477.txt.

Introduction

Minseok Son and co-authors present a 2024 arXiv paper on image recognition when training data are heavily skewed. Networks trained on such long-tailed data often miss rare classes. Past fixes rebalance by class counts but ignore that single examples also vary in difficulty within a class. The team builds both kinds of difficulty into the training loss.

Their loss adds a class-wise margin that fights frequency bias and an instance-wise margin that pushes harder on hard examples. Wider margins for hard cases sharpen the gaps between classes. The method plugs into existing approaches and, per the authors, lifts scores across standard long-tailed benchmarks.

Catalog record

CREATORMinseok Son, Inyong Koo, Jinyoung Park, Changick Kim
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15477

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Difficulty-aware Balancing Margin Loss for Long-tailed Recognition Minseok Son, Inyong Koo, Jinyoung Park, Changick Kim Abstract When trained with severely imbalanced data, deep neural networks often struggle to accurately recognize classes with only a few samples. Previous studies in long-tailed recognition have attempted to rebalance biased learning using known sample distributions, primarily addressing different classification difficulties at the class level. However, these approaches often overlook the instance difficulty variation within each class. In this paper, we propose a difficulty-aware balancing margin (DBM) loss, which considers both class imbalance and instance difficulty. DBM loss comprises two components: a class-wise margin to mitigate learning bias caused by imbalanced class frequencies, and an instance-wise margin assigned to hard positive samples based on their individual difficulty. DBM loss improves class discriminativity by assigning larger margins to more difficult samples. Our method seamlessly combines with existing approaches and consistently improves performance across various long-tailed recognition benchmarks. \AtEndPreamble \Crefname@preamble equationEquationEquations \Crefname@preamble figureFigureFigures \Crefname@preamble tableTableTables \Crefname@preamble pagePagePages \Crefname@preamble partPartParts \Crefname@preamble chapterChapterChapters \Crefname@preamble sectionSectionSections \Crefname@preamble appendixAppendixAppendices \Crefname@preamble enumiItemItems \Crefname@preamble footnoteFootnoteFootnotes \Crefname@preamble theoremTheoremTheorems \Crefname@preamble lemmaLemmaLemmas \Crefname@preamble corollaryCorollaryCorollaries \Crefname@preamble propositionPropositionPropositions \Crefname@preamble definitionDefinitionDefinitions \Crefname@preamble resultResultResults \Crefname@preamble exampleExampleExamples \Crefname@preamble remarkRemarkRemarks \Crefname@preamble noteNoteNotes \Crefname@preamble algorithmAlgorithmAlgorithms \Crefname@preamble listingListingListings \Crefname@preamble lineLineLines \crefname@preamble equationEquationEquations \crefname@preamble figureFigureFigures \crefname@preamble pagePagePages \crefname@preamble tableTableTables \crefname@preamble partPartParts \crefname@preamble chapterChapterChapters \crefname@preamble sectionSectionSections \crefname@preamble appendixAppendixAppendices \crefname@preamble enumiItemItems \crefname@preamble footnoteFootnoteFootnotes \crefname@preamble theoremTheoremTheorems \crefname@preamble lemmaLemmaLemmas \crefname@preamble corollaryCorollaryCorollaries \crefname@preamble propositionPropositionPropositions \crefname@preamble definitionDefinitionDefinitions \crefname@preamble resultResultResults \crefname@preamble exampleExampleExamples \crefname@preamble remarkRemarkRemarks \crefname@preamble noteNoteNotes \crefname@preamble algorithmAlgorithmAlgorithms \crefname@preamble listingListingListings \crefname@preamble lineLineLines \crefname@preamble equationequationequations \crefname@preamble figurefigurefigures \crefname@preamble pagepagepages \crefname@preamble tabletabletables \crefname@preamble partpartparts \crefname@preamble chapterchapterchapters \crefname@preamble sectionsectionsections \crefname@preamble appendixappendixappendices \crefname@preamble enumiitemitems \crefname@preamble footnotefootnotefootnotes \crefname@preamble theoremtheoremtheorems \crefname@preamble lemmalemmalemmas \crefname@preamble corollarycorollarycorollaries \crefname@preamble propositionpropositionpropositions \crefname@preamble definitiondefinitiondefinitions \crefname@preamble resultresultresults \crefname@preamble exampleexampleexamples \crefname@preamble remarkremarkremarks \crefname@preamble notenotenotes \crefname@preamble algorithmalgorithmalgorithms \crefname@preamble listinglistinglistings \crefname@preamble linelinelines \cref@isstackfull \@tempstack \@crefcopyformats sectionsubsection \@crefcopyformats subsectionsubsubsection \@crefcopyformats appendixsubappendix \@crefcopyformats subappendixsubsubappendix \@crefcopyformats figuresubfigure \@crefcopyformats tablesubtable \@crefcopyformats equationsubequation \@crefcopyformats enumienumii \@crefcopyformats enumiienumiii \@crefcopyformats enumiiienumiv \@crefcopyformats enumivenumv \@labelcrefdefinedefaultformats Code — https://github.com/quotation2520/dbm˙ltr Introduction \label sec:intro In recent decades, deep neural networks have demonstrated remarkable success in image recognition tasks ( Simonyan and Zisserman 2014 ; He et al. 2016 ; Szegedy et al. 2015 ) , largely due to the availability of large-scale datasets like ImageNet ( Deng et al. 2009 ) . However, real-world datasets often exhibit an imbalanced distribution, known as a long-tailed distribution, wherein a few ‘head’ classes contain a large number of samples, while numerous other classes, referred to as ‘tail’ classes, contain significantly fewer samples. This imbalance presents significant challenges: deep learning models, predominantly trained on the abundant majority classes, struggle to effectively learn features for the minority classes. As a result, models tend to underperform on these underrepresented classes, compromising their overall accuracy. Figure 1: Overview of our method. The model is trained to align samples within decision boundaries defined by adaptive margins. (1) Hard positive samples. Misclassified samples identified during training are labeled as hard positive samples. (2) Class-wise margins. Larger margins are assigned to minority classes to ensure sufficient separation from majority classes. (3) Instance-wise Margins. We propose to apply adaptive margins to hard positive samples, considering both class frequency and sample difficulty. \label fig:main Addressing class imbalance has been a focal point in long-tailed recognition (LTR) research. Existing methods have employed various strategies to rebalance the influence of different classes. Re-sampling techniques, such as oversampling ( Byrd and Lipton 2019 ) and undersampling ( Drummond, Holte et al. 2003 ) , adjust the occurrence of class samples to create a more balanced training set. Re-weighting approaches ( Cui et al. 2019 ; Menon et al. 2021 ; Ren et al. 2020 ) modify class weights or logit values to emphasize learning from difficult minority classes. For instance, the label-distribution-aware margin (LDAM) loss ( Cao et al. 2019 ) introduces larger margins for minority classes to counteract the bias towards majority classes. Despite these advances, many methods focus primarily on class-level imbalance and often overlook variations in difficulty among individual samples within each class. This oversight can lead to suboptimal performance on challenging instances, even within well-represented classes. To address this gap, we propose a novel Difficulty-aware Balancing Margin (DBM) loss that considers both instance-level difficulty and class imbalance. Unlike previous methods that primarily address class-level bias, DBM loss incorporates two key components: a class-wise margin to mitigate imbalance in class frequencies and an instance-wise margin that adapts to the difficulty of individual samples. By assigning additional margins to hard positive samples, our approach enhances class discriminability even more. \Cref fig:main illustrates the overview of our method. Here, we consider a binary classification problem where class A has more samples than class B. The decision boundary determined by the classifier is denoted by the black line in \cref fig:main(1). The misclassified samples, indicated by a two-line border, are identified as hard positive samples. Our margin loss assigns a tighter decision boundary to each sample, aiming to bring sample features closer to their class centers. First, a class-wise margin of varying sizes is applied to each sample based on its class frequency. As a result, different decision boundaries, b_{A} and b_{B} , are defined as shown in \cref fig:main(2), with the minority class experiencing a larger displacement in its decision boundary compared to the majority class. For hard positive samples, an additional instance-wise margin is applied. The final decision boundaries for the hard positive samples A1 , A2 , and B1 in \cref fig:main(3) are denoted as b_{A1} , b_{A2} and b_{B1} , respectively. Given that A1 exhibits a greater angular distance from the class center compared to A2 , A1 is assigned a larger instance-wise margin, leading to a more shift in b_{A1} relative to b_{A2} . This leads to a higher loss value for difficult samples, encouraging a denser feature distribution within each class. Our method integrates seamlessly with existing LTR techniques with negligible computational impact and demonstrates consistent performance improvements across multiple benchmarks, including CIFAR-10-LT, CIFAR-100-LT ( Cao et al. 2019 ; Kang et al. 2020b ) , ImageNet-LT ( Liu et al. 2019c ) , and iNaturalist2018 ( Van Horn et al. 2018 ) . Extensive experiments validate our design choices and showcase the effectiveness and robustness of our method. The main contributions of this paper are summarized as follows: • We propose the difficulty-aware balancing margin (DBM) loss, which effectively balances learning bias due to class imbalance and sample-level difficulty variation within a class. • Our DBM loss is compatible with various existing long-tailed recognition techniques, and incurs no significant additional computational overhead. • When combined with state-of-the-art methods, our approach demonstrates competitive performance on major long-tailed recognition benchmarks. Related Work \label sec:relwork Long-tailed Recognition Long-tailed recognition (LTR) has been extensively explored through multiple perspectives. Conventional approaches focus on rebalancing the bias introduced by imbalanced class influence during training, aiming to mitigate performance degradation for minority classes. Re-sampling methods ( Buda, Maki, and Mazurowski 2018 ; He and Garcia 2009 ) address the class imbalance by either undersampling majority classes ( Drummond, Holte et al. 2003 ; Tahir, Kittler, and Yan 2012 ) or oversampling minority classes ( Byrd and Lipton 2019 ; Ando and Huang 2017 ) . Re-weighting methods ( Cui et al. 2019 ; Cao et al. 2019 ; Ren et al. 2020 ) propose

Read the complete text