Prior2Posterior: Model Prior Correction for Long-Tailed Learning
S Divakar Bhat*, Amit More*, Mudit Soni, Surbhi Agrawal · 2024
Read Prior2Posterior: Model Prior Correction for Long-Tailed Learning by S Divakar Bhat*, Amit More*, Mudit Soni, Surbhi Agrawal (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16540.txt.
Introduction
Divakar Bhat and co-authors present a post-hoc fix for bias in skewed class data, in this 2024 paper. When training data favor common classes, models lean toward them and fail rare ones. The lean is often blamed on raw class counts, but the learned lean can differ from the raw counts.
Their method measures the model's own effective prior from its output scores, then adjusts those scores after training. No retraining is needed. Theory supports the fix for two common losses. Tests set new best marks among logit-adjustment methods on several skewed benchmarks. The trick can also audit and lift many older methods.
Catalog record
| CREATOR | S Divakar Bhat*, Amit More*, Mudit Soni, Surbhi Agrawal |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16540
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prior2Posterior: Model Prior Correction for Long-Tailed Learning S Divakar Bhat*, Amit More*, Mudit Soni, Surbhi Agrawal Abstract * * footnotetext: Equal contribution Learning-based solutions for long-tailed recognition face difficulties in generalizing on balanced test datasets. Due to imbalanced data prior, the learned a posteriori distribution is biased toward the most frequent (head) classes, leading to an inferior performance on the least frequent (tail) classes. In general, the performance can be improved by removing such a bias by eliminating the effect of imbalanced prior modeled using the number of class samples (frequencies). We first observe that the effective prior on the classes, learned by the model at the end of the training, can differ from the empirical prior obtained using class frequencies. Thus, we propose a novel approach to accurately model the effective prior of a trained model using a posteriori probabilities. We propose to correct the imbalanced prior by adjusting the predicted a posteriori probabilities (Prior2Posterior: P2P) using the calculated prior in a post-hoc manner after the training, and show that it can result in improved model performance. We present theoretical analysis showing the optimality of our approach for models trained with naive cross-entropy loss as well as logit adjusted loss. Our experiments show that the proposed approach achieves new state-of-the-art (SOTA) on several benchmark datasets from the long-tail literature in the category of logit adjustment methods. Further, the proposed approach can be used to inspect any existing method to capture the effective prior and remove any residual bias to improve its performance, post-hoc, without model retraining. We also show that by using the proposed post-hoc approach, the performance of many existing methods can be improved further. ©2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Figure 1 : We present the results on a toy dataset with the imbalance factor of 100 . The trained classifier regions are shown in purple and yellow colors and the ideal, Bayes classifier, boundaries are shown as a dashed black line in each figure. (a) Classifier boundaries are naturally biased when using a naive cross-entropy (CE) loss. (b) Using class frequencies for post-hoc correction removes the classifier bias to some extent. (c) Using proposed post-hoc correction with learned prior, boundary is adjusted very close to the optimal Bayes’ classifier. Figure 2 : We show marginal class probabilities, P(y) , for classes on CIFAR 100 -LT dataset with imbalance factor of 200 . The effective prior calculated using proposed approach and using class frequencies are shown for head classes (Many-first column), tail classes (Few-third column) and rest of the classes (Medium-middle column). One may note that model shows bias towards the head classes. Further, the bias is under represented by class frequencies for head classes and is over represented for other classes. 1 Introduction One of the primary difficulties in the case of long-tailed recognition is a distribution mismatch between training and test datasets. The training dataset is usually dominated by a large number of examples from a few classes called majority or head classes. As a result, a naive deep neural network model trained on such a dataset using simple cross-entropy loss is biased toward the dominant classes. This bias manifests in the form of higher accuracy on head classes and poor accuracy on tail classes, i.e. accuracy is directly proportional to the number of training examples for a given class. One of the early solutions proposed to mitigate the effects of such a data imbalance appears in [ 22 , 14 ] . Generally, the model bias can be removed by artificial re-balancing of the datasets and scaling of loss functions or model prediction scores. However, artificial balancing techniques such as under-sampling or oversampling the dataset can affect the quality of the learned features. In [ 12 ] , authors proposed a decoupled training strategy to first learn good quality features and then debias the model by retraining the classifier. The authors showed that vanilla empirical loss trained with instance-balanced sampling is enough to generate good-quality features. Once the features are learned, the model can be debiased by retraining the classifier with different debiasing techniques such as classifier weight scaling, weight normalization and class-balanced training. Similarly, [ 23 , 29 ] proposed logit adjusted loss and showed that retraining the classifier to introduce a relative margin between pairs of classes leads to improvements. Approaches such as these show the effectiveness of the 2-stage training paradigm widely adopted in long-tail learning. Despite using different debiasing techniques, we observe that a small bias toward the head classes remains in the final model. In this work, we propose to mitigate such a residual bias in a post-hoc manner. In [ 23 ] , authors proposed post-hoc logit adjustment when the model is trained with plain cross-entropy loss. The bias is modeled using the number of samples per class (also called class frequencies), which can re-adjust the classifier boundaries after training. However, we argue that class frequencies can not accurately represent the bias present in the trained model. Figure 2 shows the effective prior learned by a DNN model when trained on the CIFAR100-LT dataset with an imbalance factor of 200 . As shown in the figure, the model bias or a prior toward the head classes is under-represented by class frequencies, i.e. model has a stronger bias than what is indicated by class frequencies. Similarly, model bias is smaller than what is implied by class frequencies for other classes. One may attribute such behavior to the over-fitting nature of DNN models. In general, DNN model output probabilities can be considered as an estimate of the Bayesian a posteriori probabilities [ 30 ] . However, the accuracy of these estimates depend on network complexity, the amount of available training data and the effectiveness of the training algorithm i.e. convergence to the global minimum [ 30 , 14 ] . Nonetheless, using class frequencies for post-hoc debiasing will generally lead to sub-optimal improvements. Further, this approach is valid only when the model is trained with plain cross-entropy loss and instance-balanced data sampling, and combining it with complex training approaches is non-trivial. Thus, we propose to model the residual bias (also referred to as a learned prior) accurately using a posteriori class probabilities of a trained model and remove the bias in a post-hoc manner. Despite being simple, we show that the proposed approach is very effective, leads to superior performance, and can rival many state-of-the-art methods. The proposed post-hoc adjustment is statistically consistent with many training paradigms, unlike class frequency-based post-hoc adjustment, and hence can be applied to existing logit adjustment losses as well. We apply the proposed post-hoc adjustment to many existing methods without retraining the model and show that it can boost the model performance further. We summarize the specific contributions of our work: 1. We show that the effective prior of a model differs from class frequency based prior and propose how to calculate such