Single-Pass Object-Focused Data Selection

Niclas Popp, Dan Zhang, Jan Hendrik Metzen, Matthias Hein, Lukas Schott · 2024

Read Single-Pass Object-Focused Data Selection by Niclas Popp, Dan Zhang, Jan Hendrik Metzen, Matthias Hein, Lukas Schott (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10032.txt.

Introduction

Popp and co-authors tackle a budget problem in training vision systems: labeling images is expensive, so which images should be labeled first? Choosing all the images to label in one pass, before any training, is called single-pass data selection. Older methods judge whole images and often do no better than picking at random for detection and segmentation tasks.

Their method, Object-Focused Data Selection, instead uses object-level features from foundation models and makes sure every target class is well covered. Across tasks and domains it beat random choice and all rival methods. Combining its picks with automatic labels from foundation models gave the best results under tight budgets, and it also improved active learning.

Catalog record

CREATORNiclas Popp, Dan Zhang, Jan Hendrik Metzen, Matthias Hein, Lukas Schott
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10032

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Single-Pass Object-Focused Data Selection Niclas Popp, Dan Zhang, Jan Hendrik Metzen, Matthias Hein, Lukas Schott Abstract While unlabeled image data is often plentiful, the costs of high-quality labels pose an important practical challenge: Which images should one select for labeling to use the annotation budget for a particular target task most effectively? To address this problem, we focus on single-pass data selection , which refers to the process of selecting all data to be annotated at once before training a downstream model. Prior methods for single-pass data selection rely on image-level representations and fail to reliably outperform random selection for object detection and segmentation. We propose Object-Focused Data Selection (OFDS) which leverages object-level features from foundation models and ensures semantic coverage of all target classes. In extensive experiments across tasks and target domains, OFDS consistently outperforms random selection and all baselines. The best results for constrained annotation budgets are obtained by combining human labels from OFDS with autolabels from foundation models. Moreover, using OFDS to select the initial labeled set for active learning yields consistent improvements. 1 Introduction The performance of machine learning systems critically depends on the availability and quality of training data [ 60 , 3 ] . The main cost factor in the creation of a dataset is usually not the acquisition of unlabeled images, but the labeling. This is particularly true for object detection and segmentation, where instance- and pixel-level annotations are required. Labeling a single image for these tasks can take from seconds for simple cases to over 90 minutes for complex scenes [ 28 ] . This cost motivates the problem of data selection: given a pool of unlabeled data, a target task, and a fixed annotation budget, the goal is to select a subset of images for labeling that maximizes the downstream performance of models trained on it. We consider object detection and segmentation as our target tasks, for which the set of classes is known via their names. This setup is illustrated in Figure 1 . A standard approach that combines data selection simultaneously with training a downstream model is active learning. In contrast, our goal is to perform model-agnostic data selection, where data is selected before training the downstream model. The iterative setup of active learning requires multiple model training cycles , which is up to several hundred times slower than data selection in a single-pass [ 55 ] . Moreover, active learning methods require an initial dataset with labels, which is typically chosen at random (“cold start”). Using data selection methods to guide this initial choice (“warm start”) has been shown to improve downstream performance in active learning [ 7 ] . This highlights the complementary nature of data selection and active learning. Existing methods for model-agnostic and single-pass data selection rely on image-level representations and constrain the selection by the number of images to be annotated [ 64 , 27 , 55 ] . For object detection and segmentation, where the target task entails predictions on instance- or pixel-level, we demonstrate that these methods fail to consistently outperform random selection. Furthermore, the annotation budget is usually not determined per image but per annotation unit (bounding box or mask). Consequently, the annotation cost of an image varies depending on the target classes. Therefore, selecting a subset that meets a given annotation budget requires knowledge of the target classes, which is not accounted for in existing methods. Figure 1: Setup for Object-Focused Data Selection. OFDS selects a subset of images to be labeled from an unlabeled dataset given a set of object classes and an annotation budget for object-level annotations. Training downstream models with the data selected by OFDS improves over all data selection baselines and random selection (results are for an annotation budget of 5% of all annotation units for the PASCAL VOC dataset with class imbalance). The detailed experimental setup is discussed in Section 4 . In this paper, we introduce Object-Focused Data Selection (OFDS), a data selection method which aims to achieve a semantic covering of all target classes at the level of objects. While OFDS is model-agnostic and can be used for training or fine-tuning larger models, our main goal is the selection of datasets for training compact downstream models for resource-constrained environments such as autonomous driving or mobile robots, where foundation models cannot be deployed. We summarize our main contributions as follows: 1. Object-Focused Data Selection (OFDS) leverages object-level representation from foundation models and adaptive clustering to guide the data selection. Unlike prior work [ 55 , 27 , 7 , 64 ] , OFDS takes the semantic similarity on the level of objects rather than entire images into account. Furthermore, it accounts for a labeling budget given by a number of annotation units. 2. Empirical Demonstration of the Effectiveness of OFDS. We compare OFDS to 8 baselines for single-pass data selection across three tasks and four datasets. OFDS consistently improves performance, in particular in scenarios with imbalanced class distributions. 3. Combining Autolabeling with OFDS. Existing works on data selection ignore the fact that foundation models can be used to generate machine-generated autolabels that could be potentially replace human labels. We first reiterate that autolabels can rival human annotations only on simple datasets but fall short as task complexity increases. Subsequently, we propose a holistic training framework that integrates autolabels with OFDS and yields the best performance even for very small annotation budgets. 4. Warm starting active learning. Active learning methods require a small initial labeled set which is typically chosen randomly (“cold start”). We show that using OFDS for this initial selection consistently improves performance over random selection and a recent baseline [ 7 ] . 2 Related Work Single-Pass Data Selection refers to the process of selecting the entire subset of data to be annotated at once before training a downstream model. Recent methods [ 55 , 27 , 7 , 64 ] use a single or fixed number of representations per image and assume that the annotation cost is solely based on the number of images without considering the target classes. However, for tasks such as object detection and segmentation, the cost of annotating an image depends on the number of objects to be labeled, which in turn depends on the target classes. Therefore, the target classes are practically always required in order to align the data selection with a given annotation budget defined at the level of individual annotation units (bounding boxes or masks). Active Learning. Active learning aims to select the most informative instances from an unlabeled pool given the current state of a model which is being trained [ 38 ] . There are several key distinctions between single-pass data selection and active learning. First, the selected data in active learning is specific to the trained model and the selection process is done sequentially, which can further increase the overall cost for the annotation process (see App. K ). In contrast, OFDS is independent of the downstream model being trained, and the selection is performed "passively" before training. Second, most active learning methods for object-level prediction tasks are specific to single tasks such as object detection [ 57 ] or semantic segmentation [ 34 , 21 , 24 ] while

Read the complete text