Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentation
Seongsu Ha, Chaeyun Kim, Donghwa Kim, Junho Lee, Sangho Lee, Joonseok Lee · 2024
Read Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentation by Seongsu Ha, Chaeyun Kim, Donghwa Kim, Junho Lee, Sangho Lee, Joonseok Lee (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01494.txt.
Description
Abstract Referring Image Segmentation is a comprehensive task to segment an object referred by a textual query from an image. In nature, the level of difficulty in this task is affected by the existence of similar objects and the complexity of the referring expression. Recent RIS models still show a significant performance gap between easy and hard scenarios.
Catalog record
| CREATOR | Seongsu Ha, Chaeyun Kim, Donghwa Kim, Junho Lee, Sangho Lee, Joonseok Lee |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01494
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentation Seongsu Ha, Chaeyun Kim, Donghwa Kim, Junho Lee, Sangho Lee, Joonseok Lee Abstract Referring Image Segmentation is a comprehensive task to segment an object referred by a textual query from an image. In nature, the level of difficulty in this task is affected by the existence of similar objects and the complexity of the referring expression. Recent RIS models still show a significant performance gap between easy and hard scenarios. We pose that the bottleneck exists in the data, and propose a simple but powerful data augmentation method, Negative-mined Mosaic Augmentation (NeMo). This method augments a training image into a mosaic with three other negative images carefully curated by a pretrained multimodal alignment model, e.g. , CLIP, to make the sample more challenging. We discover that it is critical to properly adjust the difficulty level, neither too ambiguous nor too trivial. The augmented training data encourages the RIS model to recognize subtle differences and relationships between similar visual entities and to concretely understand the whole expression to locate the right target better. Our approach shows consistent improvements on various datasets and models, verified by extensive experiments. 1 Introduction Figure 1 : Diverse visual and linguistic challenges of referring scenarios. In (a), query (1) demands discernment among three road signs, while query (2) involves identifying a “woman”, relatively easier due to a single instance. NeMo, our method, in (b) uses similar negative images to generate a mosaic. Query (2) becomes harder as the augmented image contains additional instances of “woman” ( e.g. , women standing or sitting), and thus “in front of the wall” becomes crucial hint to solve the problem. Referring Image Segmentation (RIS) is a fundamental task in computer vision that aims to segment objects described in a natural language expression within a given scene. Central to RIS is not merely the visual recognition of objects but also the intricate understanding of the interrelationships among these objects, interpreted through linguistic cues. Each RIS problem often requires a different level of multimodal understanding capabilities, depending on visual ambiguity as well as linguistic complexity. For instance, having visually similar objects to the referent in an image complicates locating and identifying the correct object. In such a case, precise comprehension of the referring expression becomes a key to find the right target. Dataset RefCOCO RefCOCO+ G-Ref # Images 19,994 19,992 26,711 # Ref. Exp. 142,209 141,564 85,474 Query length 3.61 3.53 8.43 Obj. per query 1.76 1.67 3.03 Table 1 : Statistics of representative Referring Image Segmentation (RIS) Datasets mIoU oIoU Models Easy Hard Easy Hard LAVT [ 50 ] 78.26 54.61 79.16 47.40 CRIS [ 45 ] 76.89 52.97 78.81 43.20 CGFormer [ 43 ] 79.86 61.22 79.95 53.27 Table 2 : mIoU & oIoU on 100 easy and hard samples from G-Ref UMD test set Figure 2 : Data samples from RIS benchmarks and augmented samples using our NeMo. RefCOCO and RefCOCO+ are characterized by relatively easier scenarios with simple referring expressions, whereas G-Refs encompass more challenging sets. Fig. 1 (a) illustrates varying degrees of difficulties even within the same image. The sentence (1) is relatively harder, since three road signs exist and the model needs to find the target by precisely understanding the entire phrase. The expression (2), on the other hand, is relatively easier, since there is only one woman, so one can easily find her regardless of the rest of the phrase. Many existing RIS datasets, however, have not been created considering such challenge levels; rather, many examples can be solved by simply finding an object corresponding to the referred class. RefCOCO and RefCOCO+ [ 54 ] , for example, contain easier scenarios with less visual ambiguity and linguistic complexity. On the contrary, G-Ref [ 36 ] is considered harder. As in Tab. 1 and Fig. 2 , it contains more objects in each image on average, and queries are relatively longer. The level of difficulty significantly varies even within the same dataset. To illustrate, we manually select 100 easy and hard samples from the G-Ref UMD test set. We select ‘easy’ samples containing only a single object per its category, straightforward to identify the target without ambiguity. For instance, in the third image in Fig. 2 , there is only a single woman holding a glass, so just finding ‘a woman’ will suffice. Conversely, ‘hard’ examples contain multiple objects within the same category, necessitating a detailed perception to distinguish the intended target. Specific easy and hard examples are provided in Appendix 0.A . Tab. 2 shows a significant performance gap between the easy and hard samples by recent RIS models. This indicates that they are capable of picking the right object without ambiguity, but they tend to lack in understanding delicate meaning in the referring expressions and using them to distinguish multiple objects in the same (or similar) categories. To further improve the RIS performance, this observation reveals that we may need to revisit if the models have been provided with sufficiently difficult training data to learn from. Given this problem landscape, we aim to improve the performance by tackling the data part of the training. Specifically, we postulate that amplifying the exposure of the models to challenging examples at training could fortify their capability to understand the subtle dynamics between visual and linguistic components. Such complexity often arises when multiple objects, potentially of the same class, coexist within an image, encouraging the model to fully understand both the scene and the given referring expression. However, manually labeling such ‘hard’ data examples is prohibitively expensive. Recognizing the key factors behind the difficulty and quality of RIS training data, we introduce a simple but universally applicable data augmentation method, Negative-mined Mosaic Augmentation (NeMo). Inspired by the mosaic augmentation in YOLO v4 [ 3 ] , NeMo augments each training image by combining it with three other images in a 2\times 2 formation, showing four times more objects on average. However, NeMo differs from the previous method in that the extra three images are not chosen at random, but are carefully selected to create a properly challenging training example. Specifically, we propose considering relevance between the referring expression and candidate images, measured by a cross-modal retrieval model, e.g. , CLIP. To build a mosaic, our method selects negative images containing objects from the same or similar category to the referred object, retrieved based on the relevance to the referring expression. Augmented mosaic images mimic challenging referring examples as in Fig. 1 (b), encouraging the model to learn subtle visual differences and to concretely understand the given referring expression to better locate the target. One might concern if combining similar images may create false positives, where the correct object in an image becomes invalid due to the objects in the other quadrants. We study the possibility and impact of such false positives, and discover that it is indeed critical for the mosaic to have the right level of difficulty to be maximally effective. Based on our observations, we present a strategic retrieval process to make the mosaic neither too hard nor too easy. From extensive experiments with five state-of-the-art RIS models, we verify that NeMo consistently improves performance across all