SimLTD: Simple Supervised and Semi-Supervised Long-Tailed Object Detection
Phi Vu Tran · 2024
Read SimLTD: Simple Supervised and Semi-Supervised Long-Tailed Object Detection by Phi Vu Tran (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20047.txt.
Introduction
Phi Vu Tran presented a simple recipe in 2024 for finding rare objects in images. Real datasets are long-tailed: a few common classes have many examples, while most classes have barely any, and detectors starve on the rare ones. Earlier fixes leaned on huge labeled databases from outside, which many real projects cannot use, or added complex meta-learning machinery.
SimLTD works in three plain steps: train on the abundant classes, transfer to the scarce ones, then fine-tune on a balanced sample of both. Optional unlabeled images, cheap to gather, can join through semi-supervised learning. On the hard LVIS benchmark the framework set new records in both modes. Simplicity is the point: no outside labels, no distillation tricks.
Catalog record
| CREATOR | Phi Vu Tran |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-20047
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SimLTD: Simple Supervised and Semi-Supervised Long-Tailed Object Detection Phi Vu Tran Abstract While modern visual recognition systems have made significant advancements, many continue to struggle with the open problem of learning from few exemplars. This paper focuses on the task of object detection in the setting where object classes follow a natural long-tailed distribution. Existing methods for long-tailed detection resort to external ImageNet labels to augment the low-shot training instances. However, such dependency on a large labeled database has limited utility in practical scenarios. We propose a versatile and scalable approach to leverage optional unlabeled images, which are easy to collect without the burden of human annotations. Our SimLTD framework is straightforward and intuitive, and consists of three simple steps: (1) pre-training on abundant head classes; (2) transfer learning on scarce tail classes; and (3) fine-tuning on a sampled set of both head and tail classes. Our approach can be viewed as an improved head-to-tail model transfer paradigm without the added complexities of meta-learning or knowledge distillation, as was required in past research. By harnessing supplementary unlabeled images, without extra image labels, SimLTD establishes new record results on the challenging LVIS v1 benchmark across both supervised and semi-supervised settings. 1 Introduction The task of detecting, localizing, and classifying object instances from image and video is a long-standing problem in computer vision. Recent years have seen unprecedented progress on modern object detection systems, mostly driven by powerful neural architectures. Much of this success is measured on the relatively balanced, small-vocabulary benchmarks like PASCAL VOC [ 13 ] and MS-COCO [ 22 ] . When evaluated on a more complex and imbalanced dataset with a much larger vocabulary, however, the same models exhibit a considerable drop in detection accuracy [ 15 ] . Figure 1 : A survey comparing our SimLTD to the state of the art for long-tailed detection. We combine unlabeled data with an intuitive multi-stage training strategy to deliver the best overall performance while also optimizing for accuracy on rare classes. Our simple approach achieves superior results on the challenging LVIS v1 benchmark without requiring auxiliary image-level supervision. Figure 2 : The motivation to our approach. Left: We propose an improved head-to-tail model transfer framework for long-tailed detection by incorporating unlabeled images in both representation and transfer learning stages. Right: While it is possible to find more samples of LVIS instances from ImageNet, such an auxiliary database may not exist in another scenario like aerial imagery. Our framework does not depend on using extra image-level labels to advance long-tailed detection, which expands the applicability of our approach beyond LVIS. This paper explores ways to enhance the capability of commodity detection systems, with a particular evaluation emphasis on the challenging large-vocabulary LVIS benchmark [ 15 ] . LVIS represents a realistic application, a scenario in which object classes follow a natural long-tailed distribution. It is in the tail that most data-hungry models struggle with performance, a distribution characterized by many rare classes having as few as a single training exemplar. Figure 1 plots several recent state-of-the-art methods to address the extreme disparity of class distributions in long-tailed detection (LTD). One promising direction is to divide the problem into multiple manageable parts to help alleviate the severity of the class imbalance. LST [ 19 ] proposes to segment the overall dataset into seven phases, each phase containing progressively smaller but balanced data samples, and train a model in an incremental manner via network expansion and knowledge distillation. While LST adopts innovative ideas from class-incremental learning [ 4 , 31 , 40 ] , the method is overly complex with the maintenance of many sub-parts. Moreover, the method requires numerous stages of knowledge transfer that can lead to catastrophic forgetting and result in an inferior solution [ 12 ] . Another interesting direction is to leverage external data to augment the training instances in LVIS. The intuition is that while the rare objects may not appear frequently in natural scenes, they can be found in abundance from object-centric sources such as ImageNet [ 9 ] . By training on the union of scene-centric LVIS and object-centric ImageNet images, these weakly supervised methods [ 45 , 49 , 27 ] overcome the skewed class distribution by enriching the tail instances with additional whole-image labels. Although appealing, this approach places a hard requirement on having access to a large database of \sim 14M labeled images across \sim 21K object classes, an assumption not necessarily suitable for many practical settings outside of LVIS. Consider Figure 2 as an example for aerial imagery; there is no comparable object-centric dataset available to match the semantic concepts of “track field”, “roundabout”, or other categories in DOTA [ 11 ] . Moreover, for an industry application focusing on bespoke concepts beyond general objects, building a new object-centric dataset to supplement the main training dataset can be costly and time-consuming. There is also the open question of how to handle the genuinely rare classes ( e.g . , a rare fish species), which are strictly limited in observation and cannot be collected in more quantity from the open Internet. Can we still advance long-tailed detection without additional hand-labeling? We propose a simple and scalable framework, aptly named SimLTD, to answer this question. We deconstruct the LTD problem into three stages consisting of (i) pre-training on head classes, (ii) transfer learning on tail classes, and (iii) fine-tuning on a small sample composed of both head and tail classes. Unlike previous research on multi-stage training [ 19 , 12 ] , our framework allows for the optional use of unlabeled images to further boost LTD. We learn with supplementary unlabeled data in a semi-supervised manner via pseudo-labeling, and therefore do not explicitly rely on any additional instance-level or image-level supervision. Our work addresses several challenges associated with the head-to-tail model transfer paradigm [ 18 , 39 ] . For one, we find that the vanilla transfer of data-rich head representations to data-scarce tail classes does not yield sufficient performance improvement because the distribution of head classes is also skewed. Moreover, the naïve application of unlabeled data can aggravate the model’s inductive bias because the unlabeled samples may follow a similar long-tailed distribution. In such case, the trained model will tend to generate more pseudo labels for the head classes, resulting in a larger degree of pseudo-class imbalance. We overcome these obstacles by incorporating data augmentations specifically designed to mitigate class imbalances in both head and tail training stages. Extensive experiments in § 4.3 reveal that stronger pre-trained models on head classes, with and without unlabeled images, transfer well to the tail classes, leading to a desirable solution for long-tailed detection. Main Contributions (1) We present SimLTD, a general framework for effective supervised and semi-supervised LTD with unlabeled images. (2) The design of SimLTD is straightforward and intuitive, and is compatible with a range of backbones and detectors based on both classical convolutional and modern transformer architectures. (3) When put to the test against competing methods on the challenging LVIS v1 benchmark, SimLTD demonstrates excellent performance and scalability by establishing new state-of-the-art results with compelling margins. We hope SimLTD serves as a strong baseline for future research to tackle realistic long-tailed problems in