Active Learning with Context Sampling and One-vs-Rest Entropy for Semantic Segmentation
Fei Wu, Pablo Marquez-Neila, Hedyeh Rafii-Tari, Raphael Sznitman · 2024
Read Active Learning with Context Sampling and One-vs-Rest Entropy for Semantic Segmentation by Fei Wu, Pablo Marquez-Neila, Hedyeh Rafii-Tari, Raphael Sznitman (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06470.txt.
Introduction
Fei Wu and colleagues improved how training data for image segmentation get labeled in 2024. Drawing outlines for every class in every image is the bottleneck of semantic segmentation, above all in specialist fields. Active learning helps by asking humans to label only the most informative patches, but patch selectors tend to ignore boundary pixels, exactly where segmentation succeeds or fails.
Their method, OREAL, aggregates pixel-level uncertainty within a patch by its maximum rather than its average, so a patch containing one contested boundary scores as valuable. They also introduce one-vs-rest entropy, an uncertainty score computed class by class that quietly balances the classes as the labeled set grows, preventing common classes from crowding out rare ones. Experiments across several datasets and model architectures supported both design choices, yielding labeled sets that train better segmenters per hour of human effort.
Catalog record
| CREATOR | Fei Wu, Pablo Marquez-Neila, Hedyeh Rafii-Tari, Raphael Sznitman |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06470
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Active Learning with Context Sampling and One-vs-Rest Entropy for Semantic Segmentation Fei Wu, Pablo Marquez-Neila, Hedyeh Rafii-Tari, Raphael Sznitman Abstract Multi-class semantic segmentation remains a cornerstone challenge in computer vision. Yet, dataset creation remains excessively demanding in time and effort, especially for specialized domains. Active Learning (AL) mitigates this challenge by selecting data points for annotation strategically. However, existing patch-based AL methods often overlook boundary pixels’ critical information, essential for accurate segmentation. We present OREAL , a novel patch-based AL method designed for multi-class semantic segmentation. OREAL enhances boundary detection by employing maximum aggregation of pixel-wise uncertainty scores. Additionally, we introduce one-vs-rest entropy , a novel uncertainty score function that computes class-wise uncertainties while achieving implicit class balancing during dataset creation. Comprehensive experiments across diverse datasets and model architectures validate our hypothesis. 1 Introduction Multi-class image segmentation is one of the main problems in computer vision, with significant implications for a wide range of applications, including medical imaging, autonomous driving, and environmental monitoring. However, annotating segmentation maps in images is a time-consuming and costly task, especially for applications that require domain-specific knowledge. In an effort to alleviate this annotation burden, active learning (AL) has been proposed due to its objective of training the best possible machine learning model using the least amount of annotations collected iteratively. With its potential benefits, AL methods for image segmentation have been developed over the last few years [ 13 , 39 , 37 , 28 , 32 , 8 , 36 , 31 , 16 , 9 , 11 , 33 , 25 , 26 , 21 , 17 , 35 , 19 ] . Figure 1 : Context sampling with patch-based active learning for semantic segmentation. Mean aggregation (left) ignores patches at the boundaries of the object of interest, while maximum aggregation (right) encourages the sampling of these boundaries. This simple modification of the aggregation function resulted in an improvement of 3 percentage points in mIoU. Broadly, AL methods for image segmentation can be categorized into two: those that sample images [ 13 , 37 , 39 , 28 , 32 , 35 ] and those that sample image patches [ 8 , 36 , 31 , 11 , 26 , 21 , 17 , 33 , 9 , 16 , 25 , 19 ] . Image-level sampling methods identify which images from an unlabeled set are most beneficial to annotate, then requiring a human annotator to provide labels for each pixel of the selected images. On the other hand, patch sampling methods identify which patches ( i.e . , groups of pixels) are more beneficial to annotate across images in the unlabeled dataset, and require a human annotator to provide labels for the pixels of the selected patches. In general, patch-based methods have been shown to perform better, largely due to reduced annotation times [ 26 ] . Patch-sampling methods are more efficient than image sampling in part because they eliminate the need to annotate large amounts of uninformative background pixels. However, not all background pixels are uninformative. Empirical evidence suggests that the background pixels that are in close proximity to the object of interest are particularly beneficial in enhancing the quality of the predicted segmentations, as object boundaries are often the most challenging areas. Interestingly, existing patch-based methods do not prioritize the sampling of the background patches that correspond to these informative boundary pixels. In order to make patch-based methods aware of boundary areas, a simple but crucial observation is noting that boundary pixels often exhibit extreme statistics that differ significantly from those observed in non-boundary pixels. For example, the magnitude of the image gradient has been widely used in classical computer vision for boundary detection, as it is locally maximized in boundary pixels. Similarly, uncertainty-based scores, commonly used in AL, often reach their maximum value in boundary pixels, as these areas tend to be the most challenging. However, when scoring patches for AL, existing patch-based methods simply average the scores of the individual pixels within each patch. This averaging process diminishes the influence of boundary pixels, rendering the patch scores more uniform and thus less effective at identifying patches with informative pixels. We claim that extreme values, and not average values, are critical to properly describe the informativeness of patches. By just substituting the mean aggregation with a maximum aggregation of pixel-wise uncertainty scores, we achieved a better sampling of boundary patches (Fig. 1 ), which in turn resulted in improved segmentation quality. Therefore, we propose using aggregation functions that are sensitive to extreme values inside each patch as a simple but effective manner of improving patch-based AL methods. While maximum aggregation improves boundary sampling, selecting patches with a simple uncertainty score function ( e.g . , entropy) does not guarantee optimal results in the context of AL for multi-class semantic segmentation. Besides maximizing uncertainty, a second important criterion is class balancing of the labeled set, as it is well-known that an imbalanced labeled set can have a significant negative impact in a multi-class setting. This has led numerous works to try to address this challenge by combining model predictions with uncertainty measures [ 6 , 2 , 38 , 7 , 23 , 5 , 3 , 8 , 36 , 17 ] . Instead, we introduce a novel uncertainty score function, namely one-vs-rest entropy , that leverages class-wise uncertainties to implicitly incorporate class-balancing. Finally, we propose a new patch-based AL method OREAL for multi-class semantic segmentation. OREAL combines maximum aggregation for improved boundary sampling and the one-vs-rest entropy uncertainty score for implicit class balancing. We support our claims with an extensive collection of experiments using four datasets, three architectures (including a vision transformer), and two labeling schemes for superpixel patches. 2 Related Work We review the most relevant work in this area. 2.1 Semantic Segmentation Semantic segmentation is a process that classifies each image pixel into a predefined image category. While several weak labeling methods exist for reducing annotation cost [ 29 , 24 , 20 , 27 ] on this task, we focus on using AL methods in this work. A plethora of AL methods for semantic segmentation have been advanced, which predominantly bifurcate into two streams: image-level sampling [ 13 , 39 , 37 , 28 , 32 , 35 ] and region-specific sampling [ 8 , 36 , 31 , 16 , 9 , 11 , 33 , 25 , 26 , 21 , 17 ] . Research delineated in [ 26 ] underscores the preeminence of employing patch-based sampling rather than whole-image sampling, with an observation that reduced patch sizes correlate with enhanced results. Although there is a subset of work that engages with non-rectangular, superpixel-based patches, its advantage over rectangular patches remains debated [ 19 ] . 2.2 Superpixel Sampling Various patch shapes are possible in patch-based methods, including rectangular patches [ 16 , 9 , 11 , 33 , 26 , 25 , 36 ] and superpixels [ 8 , 19 , 31 , 21 , 17 ] . Unlike rectangular patches, superpixels adhere to the object boundaries and group pixels that are visually similar [ 14 ] , implicitly assuming that all pixels within a superpixel share a unique label. This