Concept-Aware LoRA for Domain-Aligned Segmentation Dataset Generation

Minho Park Sunghyun Park Jungsoo Lee Hyojin Park Kyuwoong Hwang Fatih Porikli Jaegul Choo Sungha Choi † KAIST Qualcomm AI Research ‡ Kyung Hee University {m.park, jchoo}@kaist.ac.kr [email protected]{sunpar, jungsool, hyojinp, kyuwoong, fporikli}@qti.qualcomm.com · 2025

Read Concept-Aware LoRA for Domain-Aligned Segmentation Dataset Generation by Minho Park Sunghyun Park Jungsoo Lee Hyojin Park Kyuwoong Hwang Fatih Porikli Jaegul Choo Sungha Choi † KAIST Qualcomm AI Research ‡ Kyung Hee University {m.park, jchoo}@kaist.ac.kr [email protected]{sunpar, jungsool, hyojinp, kyuwoong, fporikli}@qti.qualcomm.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22172.txt.

Introduction

Minho Park and colleagues proposed Concept-Aware LoRA in 2025 to generate segmentation training datasets with text-to-image models. Labeled images are costly, so synthetic data are attractive, but fine-tuned generators often just memorize their training set and miss the target domain. Their fix is selective fine-tuning.

The method finds and updates only the weights tied to concepts needed for domain alignment, such as style or viewpoint, while leaving the model's broad knowledge intact. That yields samples that are both on-domain and fresh. On urban scene segmentation, it beat prior methods in few-shot and full training settings. It also held up better under bad weather and varied lighting in domain tests.

Catalog record

CREATORMinho Park Sunghyun Park Jungsoo Lee Hyojin Park Kyuwoong Hwang Fatih Porikli Jaegul Choo Sungha Choi † KAIST Qualcomm AI Research ‡ Kyung Hee University {m.park, jchoo}@kaist.ac.kr [email protected]{sunpar, jungsool, hyojinp, kyuwoong, fporikli}@qti.qualcomm.com
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22172

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Concept-Aware LoRA for Domain-Aligned Segmentation Dataset Generation Minho Park Sunghyun Park Jungsoo Lee Hyojin Park Kyuwoong Hwang Fatih Porikli Jaegul Choo Sungha Choi † KAIST Qualcomm AI Research ‡ Kyung Hee University {m.park, jchoo}@kaist.ac.kr [email protected]{sunpar, jungsool, hyojinp, kyuwoong, fporikli}@qti.qualcomm.com Abstract This paper addresses the challenge of data scarcity in semantic segmentation by generating datasets through text-to-image (T2I) generation models, reducing image acquisition and labeling costs. Segmentation dataset generation faces two key challenges: 1) aligning generated samples with the target domain and 2) producing informative samples beyond the training data. Fine-tuning T2I models can help generate samples aligned with the target domain. However, it often overfits and memorizes training data, limiting their ability to generate diverse and well-aligned samples. To overcome these issues, we propose Concept-Aware LoRA (CA-LoRA), a novel fine-tuning approach that selectively identifies and updates only the weights associated with necessary concepts ( e.g . , style or viewpoint) for domain alignment while preserving the pretrained knowledge of the T2I model to produce informative samples. We demonstrate its effectiveness in generating datasets for urban-scene segmentation, outperforming baseline and state-of-the-art methods in in-domain (few-shot and fully-supervised) settings, as well as in domain generalization tasks, especially under challenging conditions such as adverse weather and varying illumination, further highlighting its superiority. † † ∗ Work done during an internship at Qualcomm AI Research. † † † Corresponding author. Work done while at Qualcomm AI Research. † † ‡ Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. 1 Introduction The amount of labeled data is crucial for achieving high performance in semantic segmentation. However, collecting diverse image samples, especially in rare or complex scenarios, and providing pixel-level annotations is both labor-intensive and time-consuming. To address this data scarcity, recent studies [ 60 , 1 , 51 , 52 ] have leveraged text-to-image (T2I) generation models [ 41 , 43 , 37 ] to synthesize training data for semantic segmentation. These studies utilize the rich generative capabilities of T2I models to produce promising image-label pairs. In addition, the text-driven controllability of T2I models enables targeted generation of underrepresented distributions. This has proven particularly effective for augmenting labeled datasets, alleviating data scarcity, and generating specific distributions such as balancing class [ 47 ] or weather conditions [ 22 ] . There are two primary challenges in generating segmentation datasets using image generation models: (1) generating informative datasets beyond the original training data , and (2) achieving alignment with the target domain , as illustrated in Fig. 1 . However, prior approaches have often overlooked key aspects of these challenges. Early works on segmentation dataset generation that do not utilize pretrained T2I models [ 60 , 29 , 1 , 35 ] typically produce domain-aligned samples, as the generative models are trained solely on the target dataset. Yet, due to the lack of external knowledge, they are limited in their ability to generate informative samples beyond the training distribution. In contrast, recent studies [ 51 , 52 , 57 , 22 ] leverage pretrained T2I models such as Stable Diffusion, which are trained on large-scale datasets like LAION-5B [ 46 ] , enabling them to incorporate external information. However, these methods often use the pretrained models without task-specific fine-tuning for segmentation, which leads to poor alignment with the target domain. Figure 1 : Motivation of Concept-Aware LoRA (CA-LoRA). Pretrained T2I models generate informative images but struggle with viewpoint alignment. LoRA fine-tuning on Cityscapes enables driving-viewpoint generation but leads to overfitting to the Cityscapes style and content. We aim to learn only the desired concept ( e.g . , viewpoint) for generating domain-aligned, informative samples. To address this, we apply existing fine-tuning methods [ 20 ] to align the image domain of pretrained generative models with the target domain. However, as illustrated in Figure 1 , even LoRA-based fine-tuning tends to overfit and memorize the training data, limiting the model’s ability to generate diverse and informative samples. This overfitting arises because the model learns all concepts present in the training data, such as viewpoint, style, object shape, and layout, regardless of whether they are necessary for domain alignment. To overcome this limitation, we argue that fine-tuning should selectively focus on only the essential concepts (e.g., viewpoint or style) required for domain alignment, while preserving the pretrained T2I model’s capacity to generate informative samples. Moreover, since the relevant concepts for alignment may vary across different problem settings, we propose a flexible fine-tuning method that adapts to the necessary concepts for generating urban-scene segmentation datasets . In in-domain segmentation settings, where the source and target domains are aligned ( e.g . , clear-day conditions), the T2I model primarily needs to learn the clear-day style of the training data. In contrast, in domain generalization (DG) settings, where the target domain is unseen ( e.g . , rainy-day conditions), focusing on the driving viewpoint is often more effective, as style alignment is no longer feasible. Our method supports both settings by selectively tuning the relevant concept, enabling the generation of datasets that are both informative and well aligned with the task requirements. We propose Concept-Aware LoRA (CA-LoRA), a fine-tuning method that automatically identifies and updates only the weights of T2I models associated with the desired concepts, while keeping the remaining weights frozen to preserve the pretrained knowledge. This allows the model to learn only the specified concepts , enabling it to generate images that are both well-aligned with the target domain and more diverse and informative. For example, when the desired concept is the viewpoint , our method fine-tunes the model to learn only the viewpoint from the given dataset, allowing it to generate images with varying styles, object shapes, and layouts—capabilities that standard LoRA fine-tuned models lack, as they tend to overfit to the training style and fail to generalize beyond it, even with varied text prompts. Consequently, our fine-tuned T2I models can generate specific styles based on user input, making them particularly effective in DG settings [ 6 , 19 , 18 ] . Our analysis shows that the proposed CA-LoRA can exclusively learn the specified concepts, and we demonstrate the effectiveness of the generated datasets for urban-scene segmentation by comparing them with state-of-the-art dataset generation methods [ 51 , 22 , 4 , 3 ] in both in-domain and domain generalization settings. Our contributions are threefold: 1. We propose Concept-Aware LoRA, a novel fine-tuning method that selectively identifies and updates only the weights related to the necessary concepts, reducing overfitting and preserving pretrained knowledge. 2. Concept-Aware LoRA enables T2I models to generate well-aligned and informative samples beyond training data. This addresses data scarcity by generating image-label pairs from underrepresented distributions ( e.g . , adverse weather), improving segmentation performance. 3. Our method achieves state-of-the-art performance across various settings, including a +2.30% mIoU improvement in few-shot and +1.34% in fully supervised settings on Cityscapes, as well as an average gain of +1.53% mIoU on domain generalization benchmarks. Figure 2 : Overview of our framework for generating an urban-scene segmentation dataset by learning the Cityscapes viewpoint. The process consists of four stages: (1) identifying sensitive weights for a specific concept, (2) selectively fine-tuning them with LoRA, (3) training a label generator using features from T2I model, and (4)

Read the complete text