Leveraging Contrastive Learning for Semantic Segmentation with Consistent Labels Across Varying Appearances
J. Montalvo, R. Alcover-Couso, P. Carballeira, A. Garcia-Martin, J. SanMiguel, M. Escudero-Viñolo · 2024
Read Leveraging Contrastive Learning for Semantic Segmentation with Consistent Labels Across Varying Appearances by J. Montalvo, R. Alcover-Couso, P. Carballeira, A. Garcia-Martin, J. SanMiguel, M. Escudero-Viñolo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16592.txt.
Introduction
Javier Montalvo and his five co-authors, publishing in 2024, work on teaching computers to label every pixel of a street scene, and to keep the labels steady when the weather changes. Rain, fog, and snow can fool such systems. The team built a synthetic dataset of city scenes rendered under many weather conditions. Each version of a scene shares the same pixel-perfect ground truth.
Their method uses those matched versions as a training signal. It forces the system's internal features for the same scene to agree across weather. Tests showed clear gains on several measures of feature alignment. The team also studied how much synthetic data to generate, balancing volume against variety. They present the dataset and method as a new template for adapting vision systems to shifting conditions, a key step for self-driving cars and city cameras.
Catalog record
| CREATOR | J. Montalvo, R. Alcover-Couso, P. Carballeira, A. Garcia-Martin, J. SanMiguel, M. Escudero-Viñolo |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16592
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Contrastive Learning for Semantic Segmentation with Consistent Labels Across Varying Appearances J. Montalvo, R. Alcover-Couso, P. Carballeira, A. Garcia-Martin, J. SanMiguel, M. Escudero-Viñolo Abstract This paper introduces a novel synthetic dataset that captures urban scenes under a variety of weather conditions, providing pixel-perfect, ground-truth-aligned images to facilitate effective feature alignment across domains. Additionally, we propose a method for domain adaptation and generalization that takes advantage of the multiple versions of each scene, enforcing feature consistency across different weather scenarios. Our experimental results demonstrate the impact of our dataset in improving performance across several alignment metrics, addressing key challenges in domain adaptation and generalization for segmentation tasks. This research also explores critical aspects of synthetic data generation, such as optimizing the balance between the volume and variability of generated images to enhance segmentation performance. Ultimately, this work sets forth a new paradigm for synthetic data generation and domain adaptation. 1 Introduction A key challenge in semantic segmentation is the high human and economic costs of annotating training data, which results in limited amounts of annotated real data [ 1 , 2 , 3 , 4 ] . To address this issue, two major research trends have emerged: Domain Adaptation (DA) and Domain Generalization (DG). These techniques leverage a source annotated domain (often synthetic images) to train models to effectively perform on a different but related target domain [ 5 , 6 , 7 , 8 , 9 ] . This effective translation of knowledge is achieved by learning domain-agnostic features through methods like self-training [ 7 , 10 ] , style transfer [ 11 , 12 ] , or feature alignment [ 13 , 14 ] . Despite impressive advancements in DA – generally performed in an Unsupervised fashion (UDA) – and DG for semantic segmentation, its use still underperforms compared to the ones trained with labeled target data [ 15 , 16 , 17 ] . There is theoretical room for improvement, since DA and DG achieve performance comparable to supervised methods in other tasks, such as classification [ 18 , 9 , 19 ] . In classification, feature-level domain adaptation techniques, like feature alignment, are commonly used. However, in dense prediction tasks, such as semantic segmentation, feature-level techniques are largely overlooked, in favor of output-level domain adaptation approaches, such as self-training techniques, as illustrated in Figure 1 . The primary reason for this discrepancy is that while the overall category of an image remains consistent across domains, individual pixels in semantic segmentation heavily depend on their surrounding context. Consequently, aligning features between pixels of different images can constrain the spatial learning capabilities of the model, leading to suboptimal performance. Figure 1 : Visual summary of Domain Adaptation research across three tasks broadly classified into three non-exclusive categories: input domain adaptation, feature-level domain adaptation, and output domain adaptation. Around 1000 papers were scrapped and analyzed from 2018-2023. Moreover, current UDA and DG techniques for semantic segmentation significantly underperform when used to transfer to a target scenario with harsh weather conditions [ 20 , 21 , 22 ] , limiting real-world applications like autonomous driving impractical. This underperformance can be attributed to existing synthetic urban scene datasets predominantly focusing on daylight conditions, neglecting the variability introduced by adverse weather [ 1 , 23 , 24 ] . To address the aforementioned gap, we propose a novel generation protocol that includes multiple views of each scene under different weather conditions, thereby providing semantically equivalent images to facilitate feature alignment and promote robustness across domains. Additionally, we propose a UDA and DG method to leverage the alternative versions of the semantically equivalent set of images by enforcing similar features across different weather conditions. This strategy leads to models that outperform widely-used datasets such as Synthia for DG by up to + 40\% mIoU, and up to + 13\% across four UDA methods. These benefits are extensively validated across three different alignment metrics. This work explores three fundamental questions about synthetic dataset generation: • Q1: For synthetic data generation, what is more effective: creating a larger quantity of images, or fewer images with higher variability? A1: As demonstrated in Section 4 , generating fewer images with greater variability leads to better-performing models than training with larger but less diverse datasets. • Q2: How can the variability of generated images be leveraged for semantic segmentation? A2: In Section 3 , we propose a method that leverages ground-truth information to guide feature alignment between images with varying appearances. The effectiveness of this approach is validated through experiments in Section 4 . • Q3: Which approach better improves target domain performance: matching target domain appearance or larger variability in the source domain? A3: As shown in Section 4 , source domains with extensive appearance variability lead to better target performance than training with datasets tailored to match target domain appearance. Overall, our work paves a new path for UDA and DG-based semantic segmentation through a novel synthetic data generation paradigm that enables direct application of classification-derived alignment techniques, enhancing model adaptability and performance in challenging conditions. The rest of the paper is organized as follows: Section 2 reviews the related work on synthetic data generation approaches, focusing on methods that utilize virtual environments or leverage generative deep learning models. Additionally, we outline current techniques and strategies for effective training using synthetic data. In Section 3 , we present our synthetic data generation protocol and introduce the proposed method to effectively leverage the generated data in UDA and DG scenarios. Section 4 experimentally validates this approach, demonstrating the effectiveness of the synthetic data generation protocol in combination with the proposed training scheme for the exploitation of such data variability in UDA and DG applications. Finally, Section 5 closes the paper with the conclusion remarks. 2 Related Work 2.1 Synthetic Dataset Generation Synthetic datasets offer several advantages over real data. They can be significantly more cost-effective than collecting and labeling large volumes of real data, especially when utilizing pre-existing data generation methods [ 1 , 23 , 25 ] . Additionally, they allow for data gathering in controlled environments and enable the collection of data for rare or dangerous events that would otherwise be difficult or hazardous to capture [ 26 , 27 ] . As deep learning models often require vast amounts of data for training, new tools for generating synthetic datasets are continually under development, ranging from fully controllable virtual environments to the more recent use of generative AI for creating synthetic data. Dataset Generation Using Generative Models. Diffusion models [ 28 ] , and Stable Diffusion [ 29 ] in particular, have recently gained traction in the field of generative AI due to their ability to produce high-quality images by modeling the data distribution through a series of iterative refinements. However, two major challenges persist in automatically annotating diffusion-generated images for semantic segmentation: first, accurately identifying the semantic labels present in the image, and second, spatially locating elements from each class with pixel-perfect alignment to the semantic labels. The probabilistic nature of diffusion models prioritizes plausibility and coherence over exact pixel-level accuracy, making this alignment an open research challenge. This misalignment often results in noisy labels that degrade the performance of segmentation models. While recent