Multi-Modal Guided Multi-Source Domain Adaptation for Object Detection Journal: Knowledge-Based Systems

Sangin Lee, Seokjun Kwon, Jeongmin Shin, Namil Kim, Yukyung Choi · 2026

Read Multi-Modal Guided Multi-Source Domain Adaptation for Object Detection Journal: Knowledge-Based Systems by Sangin Lee, Seokjun Kwon, Jeongmin Shin, Namil Kim, Yukyung Choi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13140.txt.

Introduction

Lee and four co-authors present MS-DePro, a multi-source domain adaptation detector that uses depth maps and text, both domain-agnostic inputs, to handle target domains unlike training data. Existing methods learn domain-agnostic features from domain-specific RGB images while keeping domain-specific information in the feature map. MS-DePro instead generates domain-agnostic region proposals from depth for localisation and aligns learnable text embeddings for classification. It reaches state-of-the-art performance on multi-source adaptation benchmarks.

Catalog record

CREATORSangin Lee, Seokjun Kwon, Jeongmin Shin, Namil Kim, Yukyung Choi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61915
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13140

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Modal Guided Multi-Source Domain Adaptation for Object Detection Journal: Knowledge-Based Systems Sangin Lee, Seokjun Kwon, Jeongmin Shin, Namil Kim, Yukyung Choi Abstract General object detection (OD) struggles to detect objects in the target domain that differ from the training distribution. To address this, recent studies demonstrate that training from multiple source domains and explicitly processing them separately for multi-source domain adaptation (MSDA) outperforms blending them for unsupervised domain adaptation (UDA). However, existing MSDA methods learn domain-agnostic features from domain-specific RGB images while preserving domain-specific information from the domain-agnostic feature map. To address this, we propose MS-DePro: Multi-Source Detector with Depth and Prompt, composed of (1) depth-guided localization and (2) multi-modal guided prompt learning. We leverage domain-agnostic input modalities, namely depth maps and text, to encode domain-agnostic characteristics. Specifically, we utilize depth maps to generate domain-agnostic region proposals for localization and integrate multi-modal features to align learnable text embeddings for classification. MS-DePro achieves state-of-the-art performance on MSDA benchmarks, and comprehensive ablations demonstrate the effectiveness of our contributions. Our code is available on GitHub . 1 Introduction Over the past decades, object detection (OD) has witnessed remarkable progress in various challenging scenarios, including closed-set and open-world settings. However, general object detectors often struggle in target (test) domains that differ from their training source domains, as domain shifts arise from several factors such as time (day vs. night), style (real vs. synthetic), and camera artifacts (sensor, resolution). To address this issue, unsupervised domain adaptation (UDA) methods [ 45 , 9 , 26 , 2 ] have emerged, aiming to mitigate the domain gap between the labeled source data and the unlabeled target data. Despite existing UDA methods assuming a single-source domain, multiple datasets are available for training. Moreover, utilizing multiple sources can yield more robust and reliable performance [ 55 ] . However, previous studies have shown that naively combining multiple sources and performing single-source UDA performs worse than using a single source [ 55 , 51 ] . In this setting, the model struggles to learn a common latent space between multiple sources and the target domain due to the discrepancies among the source domains. To tackle this issue, multi-source domain adaptation (MSDA) for OD has been proposed [ 55 , 51 , 3 , 4 ] to mitigate interference between multiple sources by treating each source domain separately. MSDA methods typically consist of two main components: 1) a domain-agnostic network and 2) a domain-specific network. The domain-agnostic network extracts common latent features through adversarial learning with a gradient reversal layer (GRL). As the GRL-based domain-agnostic network tends to remove high-level semantic information [ 13 ] , they simultaneously learn domain-relevant knowledge in the domain-specific network. For instance, DMSN [ 55 ] and TPKP [ 51 ] employ separate subnets for each source domain, while PMT [ 3 ] and ACIA [ 4 ] develop a prototype network and attention-based instance alignment method, respectively. Figure 1: Comparison of MSDA methods. (a) Existing MSDA methods rely solely on RGB images to learn domain-agnostic features, while also encoding domain-specific features from them, which leads to a training conflict. (b) Our MS-DePro leverages additional domain-agnostic modalities—depth map and text—to encode domain-agnostic features. Although existing MSDA methods for OD have shown promising results, they have a fundamental limitation in modeling domain-agnostic and domain-specific knowledge. As shown in Figure 1 -(a), these methods first learn domain-invariant representations from domain-specific RGB images by intentionally suppressing domain-specific knowledge using GRL. From these domain-agnostic features, they then strive to encode domain-specific representations. This paradoxical process causes a training conflict between the objectives of these two networks, which impedes optimization and slows convergence. Furthermore, encoding domain-agnostic features from RGB images is fundamentally constrained, as they are highly sensitive to variations in illumination, color, and texture. To address this, we explicitly disentangle domain-specific and domain-agnostic features and extract them separately by leveraging additional domain-agnostic modalities. To this end, we consider depth maps and text (see Figure 1 -(b)). Compared to RGB images, depth maps provide more consistent representations by focusing on object shape and structure, thereby reducing reliance on domain-specific cues [ 16 ] . Additionally, the text modality conveys high-level semantic information that remains consistent across domains [ 1 ] . We argue that employing domain-agnostic modalities at the input level prevents interference from domain-specific knowledge. In this paper, we propose a novel framework termed Multi-Source Detector with Depth and Prompt (MS-DePro). Our MS-DePro consists of two main components: (1) depth-guided localization and (2) multi-modal guided text prompt learning. First, we exploit depth maps for domain-invariant object localization. To this end, we employ off-the-shelf monocular depth estimation models [ 54 , 5 ] to extract relative depth maps from input RGB images, which are only used during the training phase. The depth maps support RGB-only localization by complementing regions characterized by shape and structural cues. Next, multi-modal guided prompt learning leverages the high-level domain-agnostic characteristics of the text modality. Specifically, the domain-agnostic parts of the text prompt are enriched with features from depth maps, thus enhancing shared representations across multiple domains. Meanwhile, the domain-specific parts of the text prompt adjust text embeddings using RGB features, enabling domain-aware alignment between text and visual regions for improved object classification. Leveraging our proposed components, MS-DePro effectively encodes domain-agnostic features without relying on domain adversarial learning. The main contributions of this work are threefold: 1. We introduce MS-DePro, a novel MSDA framework for OD, which leverages auxiliary modalities, i.e., depth map and text, to explicitly encode domain-invariant representations. 2. We propose depth-guided localization and multi-modal guided prompt learning. The depth-guided localization exploits depth maps to assist object localization, and multi-modal guided prompt learning explicitly integrates domain-agnostic and domain-specific features for improved classification. 3. We conduct extensive experiments on MSDA benchmarks, and our MS-DePro achieves state-of-the-art performance. We further evaluate its generalization by extending experiments to the domain generalization setting. The remainder of this manuscript is organized as follows: Section 2 provides a brief overview of the related work. Section 3 elaborates the proposed MS-DePro and its two constituent modules. Section 4 presents extensive experimental results on the MSDA and MSDG benchmarks and thorough ablation studies. Section 5 summarizes the findings and outlines the limitations. 2 Related Work 2.1 Domain-Agnostic Representation Learning As a branch of domain adaptation, research in multi-target [ 29 ] , blended-target [ 32 ] , and source-free [ 31 , 30 ] scenarios has focused on distribution alignment to minimize domain discrepancy. These diverse branches demonstrate that effectively mitigating domain shifts fundamentally relies on capturing domain-agnostic representations. Meanwhile, some studies [ 33 , 41 , 61 ] have focused on learning causal representations to disentangle invariant features from spurious correlations through causal interventions within the RGB space, aiming to yield domain-agnostic representations. Specifically, Ouyang et al. [ 33 ] eliminated domain-specific appearance shortcuts, while Song et al. [ 41 ] and Zhou et al. [ 61 ] enforced causal separation through target style imitation and graph autoencoders, respectively. In this paper, we ensure the extraction of domain-agnostic features by leveraging depth maps and text directly at the input level, as RGB images are fundamentally domain-specific. By incorporating these inherently invariant modalities, our MS-DePro bypasses the limitations of relying solely on RGB

Read the complete text