Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Linchao Zhang, Xiaochun Cao, Qingming Huang · 2025
Read Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification by Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Linchao Zhang, Xiaochun Cao, Qingming Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01660.txt.
Introduction
Sicong Li and co-authors present a 2025 computing paper on sorting images when some classes are rare. Real data sets often have a long tail: a few classes hold most samples, and many classes hold very few. Models tend to learn the common classes well and the rare ones poorly.
Their method, Focal-SAM, builds on a training trick that seeks flat spots in the loss landscape, where small shifts do not hurt much. Prior versions either skipped the common classes or paid a high cost in extra passes. Focal-SAM gives each class its own penalty weight, gaining fine control at no extra pass cost. The authors prove a tighter bound on how well it should work, and tests on standard and large models back the claim.
Catalog record
| CREATOR | Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Linchao Zhang, Xiaochun Cao, Qingming Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01660
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Linchao Zhang, Xiaochun Cao, Qingming Huang Abstract Real-world datasets often follow a long-tailed distribution, making generalization to tail classes difficult. Recent methods resorted to long-tail variants of Sharpness-Aware Minimization (SAM), such as ImbSAM and CC-SAM, to improve generalization by flattening the loss landscape. However, these attempts face a trade-off between computational efficiency and control over the loss landscape. On the one hand, ImbSAM is efficient but offers only coarse control as it excludes head classes from the SAM process. On the other hand, CC-SAM provides fine-grained control through class-dependent perturbations but at the cost of efficiency due to multiple backpropagations. Seeing this dilemma, we introduce Focal-SAM, which assigns different penalties to class-wise sharpness, achieving fine-grained control without extra backpropagations, thus maintaining efficiency. Furthermore, we theoretically analyze Focal-SAM’s generalization ability and derive a sharper generalization bound. Extensive experiments on both traditional and foundation models validate the effectiveness of Focal-SAM. 1 Introduction In the past decades, deep learning has achieved remarkable success in various fields, including image classification ( He et al. 2016 ) , medical image processing ( Ronneberger et al. 2015 ) , and object detection ( Ren et al. 2015 ) . However, this success often relies on carefully curated, balanced datasets. In real-world scenarios, data often exhibits a long-tailed distribution, where a few categories have abundant samples while most categories contain only a small number of examples. Long-tailed learning focuses on effectively training models on such imbalanced datasets ( Zhang et al. 2023 ; Zhang et al. 2024a ) . Numerous approaches have been proposed to address this challenge, including re-sampling ( Buda et al. 2018 ) , re-balancing ( Cui et al. 2019 ; Ren et al. 2020 ; Wang et al. 2023 ) , representation learning ( Zhu et al. 2022 ; Cui et al. 2024 ) , ensemble learning ( Wang et al. 2021 ; Zhang et al. 2022 ) , and fine-tuning foundation models ( Dong et al. 2023 ; Shi et al. 2024 ) . Recently, Rangwani et al. 2022 visualized the loss landscape of different classes and observed that tail classes often suffer from saddle points. Since the loss landscape is closely related to the generalization of modern neural networks ( Keskar et al. 2017 ; Jiang et al. 2020 ) , they apply Sharpness-Aware Minimization (SAM) ( Foret et al. 2021 ) to help tail classes escape from saddle points. Later, since the original SAM operates on all classes, ImbSAM ( Zhou et al. 2023a ) excludes the head classes to better focus on flattening the landscape of the tail classes. However, when combined with popular re-balancing methods ( Cao et al. 2019 ; Kini et al. 2021 ; Menon et al. 2021 ) , this coarse-grained approach often overemphasizes the tail classes, leading to poor head and overall performance. To achieve fine-grained control, CC-SAM ( Zhou et al. 2023b ) uses class-dependent perturbation. However, the per-class perturbation requires at least C additional backpropagations, where C denotes the number of classes, making it rather computationally expensive. This raises a natural question: Can we design a method that achieves both fine-grained control and computational efficiency? Figure 1 : (a) ImbSAM applies the sharpness penalty only to tail classes, leading to a sharp loss landscape for head classes. In contrast, Focal-SAM assigns class-specific weights to the sharpness penalty, resulting in smooth loss landscapes for both head and tail classes. (b) Focal-SAM replaces per-class perturbations in CC-SAM with class-specific sharpness penalties, significantly enhancing computational efficiency while achieving better performance. Targeting this goal, we integrate the focal mechanism ( Lin et al. 2017 ) with SAM, inducing a novel approach named Focal-SAM. Specifically, we introduce the focal sharpness term, which is defined as the weighted sum of class-wise sharpness, where the weights decrease in a focal-like manner from head to tail classes. On the one hand, Focal-SAM controls the flatness of different classes in a fine-grained way, better balancing the performance between head and tail classes than ImbSAM, as shown in Fig. 1 (a). On the other hand, Focal-SAM replaces the per-class perturbations in CC-SAM with per-class sharpness penalties, making it much more efficient than CC-SAM, as illustrated in Fig. 1 (b). Furthermore, we provide an informative generalization bound based on the PAC-Bayesian theory. This bound not only decreases at a faster rate than those of SAM and CC-SAM ( \tilde{\mathcal{O}}(1/n) vs. \mathcal{O}(1/\sqrt{n}) , where n is the number of training samples) but also demonstrates the influence of the hyperparameters and trace of the Hessian. Finally, we conduct extensive experiments on various benchmark datasets to validate the effectiveness of Focal-SAM, including training ResNet models from scratch and fine-tuning the foundation model CLIP ( Radford et al. 2021 ) . The results show that Focal-SAM consistently outperforms other SAM-based methods across multiple datasets and models in long-tailed recognition tasks. Prior arts ( Zhou et al. 2022 ; Khattak et al. 2023 ; Park et al. 2024 ) have demonstrated that fine-tuning CLIP often performs well on the target domain but struggles with domain shifts. Therefore, we also assess model performance on OOD test sets when fine-tuning foundation models, referred to as long-tailed domain generalization tasks. The results indicate that Focal-SAM improves performance by approximately 0.5% \sim 4.3% when combined with baselines on OOD test sets. These further suggest that Focal-SAM can enhance generalization, leading to better performance under domain shifts. In summary, our key contributions are as follows: • Systematic studies illustrate the limitations of ImbSAM and CC-SAM. ImbSAM fails to flatten the loss landscape for head classes, while CC-SAM is highly computationally expensive. • We propose Focal-SAM, a simple yet effective method that provides fine-grained control of loss landscape and maintains computational efficiency. Theoretical analysis further offers a sharp generalization bound of Focal-SAM. • Extensive experiments validate the effectiveness of the proposed Focal-SAM, ranging from training ResNet models from scratch to fine-tuning foundation models. 2 Related Work 2.1 Long-Tailed Learning Several approaches address long-tailed learning challenges, such as re-sampling ( Buda et al. 2018 ; Wang et al. 2019b ; Liu et al. 2022 ) , re-balancing ( Cui et al. 2019 ; Ren et al. 2020 ; Wang et al. 2023 ; Wang et al. 2022 ; Han et al. 2024 ; Hou et al. 2022 ; Lyu et al. 2025 ; Yang et al. 2023b ; Yang et al. 2023a ; Yang et al. 2022 ; Zhao et al. 2024a ; Dai et al. 2023 ; Shao et al. 2023 ; Hong et al. 2024 ) , data augmentation ( Kim et al. 2020 ; Hong et al. 2022 ; Ahn et al. 2023 ; Wang et al. 2024b ; Wang et al. 2024a ) , representation learning ( Cui et al. 2021 ; Zhu et al. 2022 ; Cui et al. 2024 ; Gao et al. 2023 ; Zhang et al. 2024b ) , ensemble learning ( Wang et al. 2021 ; Zhang et al. 2022 ; Li et al. 2022 ; Aimar et al. 2023 ; Yang et al. 2024 ; Zhao et al. 2024b ) , and fine-tuning foundation models ( Dong et al. 2023