Solving the long-tailed distribution problem by exploiting the synergies and balance of different techniques
Ziheng Wang, Toni Lassila, Sharib Ali · 2025
Read Solving the long-tailed distribution problem by exploiting the synergies and balance of different techniques by Ziheng Wang, Toni Lassila, Sharib Ali (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13756.txt.
Introduction
Wang, Lassila, and Ali study how three remedies for skewed data work as a team. In long-tailed data, a few common classes swamp many rare ones, and models neglect the rare ones. The authors combine supervised contrastive learning, which tightens class clusters, a generator that synthesizes fresh examples of rare classes, and a loss that grants rare classes a wider safety margin.
Each method covers another's weakness. The generator refills the tail space that contrastive learning squeezes; the margin loss sharpens the boundaries the other two draw; and contrastive learning restores the head-class accuracy the other two sacrifice. On long-tailed benchmarks the combination lifts tail accuracy while holding the head steady, which the authors frame as synergy worth designing for.
Catalog record
| CREATOR | Ziheng Wang, Toni Lassila, Sharib Ali |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Solving the long-tailed distribution problem by exploiting the synergies and balance of different techniques Ziheng Wang, Toni Lassila, Sharib Ali Abstract In real-world data, long-tailed data distribution is common, making it challenging for models trained on empirical risk minimization to learn and classify tail classes effectively. While many studies have sought to improve long-tail recognition by altering the data distribution in the feature space and adjusting model decision boundaries, research on the synergy and corrective approach among various methods is limited. Our study delves into three long-tail recognition techniques: Supervised Contrastive Learning (SCL), Rare-Class Sample Generator (RSG), and Label-Distribution-Aware Margin Loss (LDAM). SCL enhances intra-class clusters based on feature similarity and promotes clear inter-class separability, but tends to favor dominant classes only. When RSG is integrated into the model, we observed that the intra-class features further cluster towards the class center, which demonstrates a synergistic effect together with SCL’s principle of enhancing intra-class clustering. RSG generates new tail features and compensates for the tail feature space squeezed by SCL. Similarly, LDAM is known to introduce a larger margin specifically for tail classes; we demonstrate that LDAM further bolsters the model’s performance on tail classes when combined with the more explicit decision boundaries achieved by SCL and RSG. Furthermore, SCL can compensate for the dominant class accuracy sacrificed by RSG and LDAM. Our research emphasizes the synergy and balance among the three techniques, with each amplifying the strengths of the others and mitigating their shortcomings. Our experiment on long-tailed distribution datasets, using an end-to-end architecture, yields competitive results by enhancing tail class accuracy without compromising dominant class performance, achieving a balanced improvement across all classes. 1 Introduction (a) (b) (c) (d) (e) (f) Figure 1 : (a) shows the feature distribution for long-tail recognition based on Cross-Entropy (CE), where purple triangles represent head classes and green circles represent tail classes [ 35 ] . (b) Supervised Contrastive Learning (SCL) in binary classification promotes intra-class clustering and inter-class separation, which can be regarded as a mutual repulsion (represented by green arrows) between the clusters of the two distinct classes. The shadow represents the position of the feature under CE classification, and the blue arrows indicate the trajectory of feature movement. (c) Feature distribution under the influence of SCL. Ideally, where both classes are tightly clustered, a void feature region exists between them, devoid of any feature distribution, signifying a clear separation of classes. (d) Rare-Class Sample Generator (RSG) calculates a set of class centers, and for clarity, only one class center (red star) is drawn for each class. RSG drives the features of a class towards its class center. Features with a red outline further cluster towards the class center from their position under SCL (shadowed area). (e) RSG transfers the feature displacement (black arrow) between the head class samples and the class center to tail samples, generating a new tail sample (green circle with a purple outline). RSG expands the feature space of tail samples, thereby influencing the potential decision boundary. (f) Label-Distribution-Aware Margin Loss (LDAM) calculates the margin based on the number of samples of each class, providing a larger margin, \gamma_{2} , for tail samples. The solid black line represents the adjusted decision boundary by LDAM. Computer vision has achieved significant successes and has been widely used in real-world applications due to advancements in deep convolutional neural networks (CNNs) and the availability of large and high-quality datasets. However, real-world data, in general, follows a long-tailed distribution. As a result, CNN models trained on balanced datasets struggle to perform well on skewed datasets, especially for classes with few samples (tail classes). These tail classes are of imminent importance as they usually represent critical problems that could widely benefit society, such as severe diseases, endangered species, fraud activities, etc. Therefore, training unbiased models for long-tail recognition (LTR) is of great practical significance. Traditional re-sampling and re-weighting methods fail to address LTR well [ 35 , 4 , 44 , 37 , 14 , 36 , 2 ] . Thus, some logit adjustment approaches [ 28 , 3 , 32 ] that directly adjust decision boundaries for a balanced feature distribution have been proposed and widely researched. Recent studies have improved classification accuracy using ensemble models [ 44 , 27 , 42 ] , but several studies have often introduced greater complexity. For example, decoupled learning [ 3 , 17 , 24 , 44 ] adopts a two-stage approach, focusing on representation learning and classifier training separately to optimize each without conflicts. Supervised Contrastive Learning (SCL), as detailed in [ 17 ] , offers robust capabilities for representation learning on balanced datasets. Originally designed for two-stage training, SCL’s potential in LTR is further unlocked in an end-to-end hybrid network, Hybrid-SC in [ 35 ] . The approach combines SCL for feature learning with Cross-Entropy (CE) for classification, outperforming the conventional two-stage SCL method only. The Hybrid-SC highlights the effectiveness of SCL in long-tail recognition, and the same authors also designed a prototypical supervised contrastive learning strategy (Hybrid-PSC) to save memory space. The Balanced Contrastive Learning (BCL) [ 45 ] extends SCL by integrating Class-averaging and Class-complement strategies to ensure equitable learning across both head and tail classes. The BCL leverages data augmentation to enhance representation learning. Additionally, it employs a logit adjustment classifier [ 28 ] to bias the model towards tail classes further. The state-of-the-art (SOTA) results demonstrate the efficacy and synergy of logit adjustment classification with SCL-based representation learning in LTR. Our method is based on principles from multi-task learning [ 31 ] , an end-to-end model with distinct branches to optimize SCL, Rare-Class Sample Generator (RSG) [ 34 ] , and Label-Distribution-Aware Margin Loss (LDAM) [ 3 ] individually. SCL’s performance is inherently linked to class sample sizes, favoring head classes [ 4 , 45 ] and thus limiting the representation of tail classes. In contrast, LDAM has been shown to significantly enhance the accuracy of tail classes, more so than other loss functions, at the cost of head class accuracy [ 43 ] . The integration of RSG into the model exacerbates this effect, further shifting the model attention towards tail classes [ 34 ] . Our goal is to achieve a synergy where each component capitalizes on its strengths to compensate for the limitations of the others. While SCL tends to compress tail classes into tighter feature spaces, RSG works to expand these spaces, as shown in Figure 1(e) . SCL and RSG promote more distinct class clustering with enhanced inter-class separation, as illustrated in Figures 1(b) to 1(d) , mitigating the negative impact on head classes when LDAM is applied to introduce larger margins for tail classes, as demonstrated in Figure 1(f) . Our main contributions are as follows: • In our study, by simply minimizing a weighted linear combination of losses, we identified a synergistic and compensatory relationship between SCL, RSG, and LDAM for long-tail recognition, which yielded competitive results. • Ablation experiments revealed that balancing the strengths and weaknesses of SCL, RSG, and LDAM offers an effective solution for long-tail recognition challenges. • We emphasize that when classes are distinctly separable, expanding the feature space for tail classes to boost their accuracy does not