LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups
Masih Aminbeidokhti, Subhankar Roy, Eric Granger, Elisa Ricci, Marco Pedersoli · 2025
Read LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups by Masih Aminbeidokhti, Subhankar Roy, Eric Granger, Elisa Ricci, Marco Pedersoli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10683.txt.
Introduction
Masih Aminbeidokhti and four co-authors study learning from long-tailed data, where a few head classes have many examples and many tail classes have very few. Parameter-efficient fine-tuning methods such as LoRA are known to protect tail-class accuracy on foundation models, but the authors find this comes at the cost of the head classes.
Controlled experiments on CIFAR100 identify the head-tail ratio, the proportion of head to tail classes, as a key overlooked factor in that trade-off. Their fix, LT-Soups, works in two stages. First it averages several models fine-tuned on balanced subsets to reduce head bias. Then it fine-tunes only the classifier on the full dataset to restore head accuracy. Across six benchmarks, LT-Soups achieved better trade-offs than both fine-tuning methods and traditional model soups.
Catalog record
| CREATOR | Masih Aminbeidokhti, Subhankar Roy, Eric Granger, Elisa Ricci, Marco Pedersoli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-10683
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups Masih Aminbeidokhti, Subhankar Roy, Eric Granger, Elisa Ricci, Marco Pedersoli Abstract Real-world datasets typically exhibit long-tailed (LT) distributions, where a few head classes dominate and many tail classes are severely underrepresented. While recent work shows that parameter-efficient fine-tuning (PEFT) methods like LoRA and AdaptFormer preserve tail-class performance on foundation models such as CLIP, we find that they do so at the cost of head-class accuracy. We identify the head-tail ratio, the proportion of head to tail classes, as a crucial but overlooked factor influencing this trade-off. Through controlled experiments on CIFAR100 with varying imbalance ratio ( ρ \rho ) and head-tail ratio ( η \eta ), we show that PEFT excels in tail-heavy scenarios but degrades in more balanced and head-heavy distributions. To overcome these limitations, we propose LT-Soups, a two-stage model soups framework designed to generalize across diverse LT regimes. In the first stage, LT-Soups averages models fine-tuned on balanced subsets to reduce head-class bias; in the second, it fine-tunes only the classifier on the full dataset to restore head-class accuracy. Experiments across six benchmark datasets show that LT-Soups achieves superior trade-offs compared to both PEFT and traditional model soups across a wide range of imbalance regimes. 1 Introduction In machine learning, balanced class distributions are often assumed in both theory and practice 11 ; 65 ; 25 . However, real-world datasets frequently deviate from this assumption, exhibiting severe class imbalance where a few head classes dominate and tail classes remain significantly underrepresented 51 ; 16 ; 31 . This imbalance poses a fundamental challenge: models must learn effectively from limited tail-class data while preserving overall robustness 9 . Recent advances in vision-language foundation models, particularly CLIP 39 , have introduced promising tools for addressing class imbalance. Trained on large-scale, diverse datasets, CLIP demonstrates strong robustness to distributional shifts and has become a popular backbone for long-tailed recognition 54 ; 53 ; 34 ; 50 ; 32 . Building on this, 46 achieve state-of-the-art results by applying parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) 17 and AdaptFormer 8 , in combination with logit adjustment (LA) loss 36 ; 42 , which incorporates class priors by adding a class-dependent offset to the logits. While this PEFT-based approach improves overall and tail-class accuracy, they observe that it still underperforms full fine-tuning in certain regimes. Figure 1 : Performance of baselines and LT-Soups on the CIFAR100 benchmark varying \rho and \eta . While full fine-tuning generally outperforms PEFT on head classes, PEFT demonstrates superior performance on tail classes. In contrast, our approach maintains robust accuracy across all imbalance settings, showing resilience to shifts in both the sample distribution and class structure. These observations motivate a deeper investigation into when and why PEFT is effective. To this end, we construct a controllable long-tailed benchmark based on CIFAR100 that allows systematic variation in both the sample counts across classes, quantified by the imbalance ratio ( \rho ), and the number of classes above or below a sample threshold, defined as the head-tail ratio ( \eta ). This setup enables a more fine-grained analysis of imbalance structures. Within this framework, we compare two full fine-tuning strategies against a PEFT baseline: full fine-tuning with logit adjustment (LA) and model soups 55 , which averages the weights of multiple LA-trained models initialized with different seeds and hyperparameters. Results from this benchmark confirm previous findings: on average, PEFT improves overall accuracy (80.8 vs. 81.2) and tail-class accuracy (76.4 vs. 70.2) compared to full fine-tuning, but at the cost of degraded head-class performance (87.0 vs. 84.3). A detailed breakdown in Figure 1 shows that PEFT is especially effective in tail-heavy scenarios, where rare classes dominate ( \eta\ll 1 ), but its performance declines as the class structure becomes more balanced or head-heavy, highlighting its limited robustness to shifts in class structure. This highlights a key trade-off: PEFT helps prevent overfitting and supports tail classes, but lacks adaptability in more balanced and head-heavy settings. Conversely, full fine-tuning offers stronger adaptation but requires careful regularization. Model soups offer a middle ground by averaging models trained with different seeds and hyperparameters 26 ; 24 , but our experiments show that traditional soups, built from models trained on the same imbalanced dataset, still underperform in tail-heavy cases, as they tend to overemphasize head-class performance due to the dominance of high imbalance ratios. To address these limitations, we introduce LT-Soups, a two-stage model soups framework designed to deliver robust performance across diverse imbalance scenarios, jointly characterized by the imbalance ratio ( \rho ) and the head-tail ratio ( \eta ). Unlike PEFT, which performs well primarily in tail-heavy settings, LT-Soups consistently achieves strong results across tail-heavy, balanced, and head-heavy class structures (Figure 1 ). In the first stage, LT-Soups constructs a weight ensemble by averaging multiple fully fine-tuned models, each trained on a subset exhibiting a distinct imbalance ratio, collectively spanning a spectrum of imbalance ratios. The aim is to create models that “specialize” for each of these imbalance ratios, when averaged, promote a balanced representation that performs well on both the head and the tail classes. To recover any head-class information lost during subsampling, the second stage fine-tunes only the classifier on the full dataset using class-balancing techniques. By seamlessly combining the adaptability of full-rank optimization, favouring the head-classes, and the robustness of weight ensembling for the tail-classes, LT-Soups strikes a better trade-off than PEFT and model soups, and thus bridges the head and the tail classes. Our contributions are threefold: (1) We introduce a dual-axis framework for characterizing class imbalance using both the imbalance ratio ( \rho ) and head-tail ratio ( \eta ), and show how they jointly affect performance. (2) We propose LT-Soups , a novel two-stage approach that mitigates representation bias and adapts effectively across a broad range of imbalance structures. (3) We conduct extensive experiments on five benchmark datasets and show that while existing LT methods perform well only under specific imbalance configurations, our approach consistently delivers robust, all-around performance across a wide range of imbalance scenarios. 2 Related Work Imbalanced Learning. Class imbalance has traditionally been tackled through oversampling minority classes, undersampling majority classes, or applying reweighted loss functions such as focal loss or logit adjustment (LA) 7 ; 15 ; 36 . While effective in certain settings, these techniques often struggle under overparameterized models 60 . Decoupled training frameworks further refine this process by separating representation learning and classifier training 22 ; 61 , assuming biases lie primarily in the classifier layer. However, this assumption breaks down when adapting foundation models, as full fine-tuning can lead to catastrophic forgetting and degraded generalization for both head and tail classes 37 ; 46 . Ensemble-based methods address class imbalance by combining experts trained on diverse data distributions 4 ; 48 . Examples include BBN 66 and RIDE 52 , which use architectural branching or dynamic routing, and LFME 57 , which employs group-wise distillation. Mixture-of-Experts approaches such as SADE 62 , Mdcs 64 , and DirMixE 59 merge experts trained with different logit adjustments (e.g., uniform, long-tail, inverse long-tail). Unlike these methods that require all experts at inference, LT-Soups collapses multiple fine-tuned models into a single