MCA: 2D-3D Retrieval with Noisy Labels via Multi-level Adaptive Correction and Alignment

Gui Zou, Chaofan Gan, Chern Hong Lim, Supavadee Aramvith, Weiyao Lin · 2025

Read MCA: 2D-3D Retrieval with Noisy Labels via Multi-level Adaptive Correction and Alignment by Gui Zou, Chaofan Gan, Chern Hong Lim, Supavadee Aramvith, Weiyao Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06104.txt.

Introduction

Five authors, led by Gui Zou, present a 2025 method to match flat images with 3D models even when many training labels are wrong. Cross format search is growing as 2D and 3D data pile up, but bad labels trip most methods. Past fixes split samples inside each format alone, which lets wrong labels pull the model off course.

Their method corrects labels by blending each format's past guesses, so the two views check each other. A second step aligns features at several levels to sharpen the match. On standard and noisy test sets, the method beat prior tools. The authors stress its calm under messy labels.

Catalog record

CREATORGui Zou, Chaofan Gan, Chern Hong Lim, Supavadee Aramvith, Weiyao Lin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06104

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MCA: 2D-3D Retrieval with Noisy Labels via Multi-level Adaptive Correction and Alignment Gui Zou, Chaofan Gan, Chern Hong Lim, Supavadee Aramvith, Weiyao Lin Abstract With the increasing availability of 2D and 3D data, significant advancements have been made in the field of cross-modal retrieval. Nevertheless, the existence of imperfect annotations presents considerable challenges, demanding robust solutions for 2D-3D cross-modal retrieval in the presence of noisy label conditions. Existing methods generally address the issue of noise by dividing samples independently within each modality, making them susceptible to overfitting on corrupted labels. To address these issues, we propose a robust 2D-3D M ulti-level cross-modal adaptive C orrection and A lignment framework (MCA). Specifically, we introduce a Multimodal Joint label Correction (MJC) mechanism that leverages multimodal historical self-predictions to jointly model the modality prediction consistency, enabling reliable label refinement. Additionally, we propose a Multi-level Adaptive Alignment (MAA) strategy to effectively enhance cross-modal feature semantics and discrimination across different levels. Extensive experiments demonstrate the superiority of our method, MCA, which achieves state-of-the-art performance on both conventional and realistic noisy 3D benchmarks, highlighting its generality and effectiveness. † † ∗ Corresponding author. Email: [email protected] † † address: 1 Shanghai Jiao Tong University, China 2 Monash University, Malaysia 3 Chulalongkorn University, Thailand 1 Introduction With the rapid growth of 3D data, research on its analysis and applications has gained significant progress. A key application in leveraging 3D data is 2D-3D cross-modal retrieval, which is essential for tasks such as surveillance [ 1 ] , virtual reality [ 2 ] , and robotics [ 3 ] . Compared to 2D data, 3D data exhibit complex geometric structures, high-dimensional representations, and irregular distributions, making their processing more challenging. Moreover, the rapid expansion of 3D datasets has made annotation increasingly costly and labor-intensive, leading to prevalent label noise in cross-modal retrieval tasks. Consequently, learning effectively under noisy labeling conditions has become a critical challenge in advancing 2D-3D cross-modal retrieval techniques. Previous studies [ 4 , 5 ] on cross-modal retrieval with noisy labels generally follow two steps: mitigating the adverse effects of noisy labels and bridging the heterogeneity gap between multimodal data. To address label noise, various methods such as GCE [ 6 ] , Label Smoothing [ 7 ] , and DivideMix [ 8 ] have been proposed, demonstrating effectiveness in unimodal scenarios. Meanwhile, to reduce the heterogeneity gap in multimodal data, approaches leveraging center loss [ 9 ] have been explored, aiming to align cross-modal features while minimizing intra-class variations. These methods have proven effective in learning a shared feature space when well-annotated data are available. However, conventional noise-robust learning methods [ 10 , 11 ] are primarily designed for unimodal tasks and struggle to extend to 2D-3D multimodal domains due to the substantial heterogeneity and domain gap between 2D and 3D data. Meanwhile, traditional supervised cross-modal alignment approaches [ 12 ] are highly sensitive to label noise and prone to overfitting on corrupted annotations. Recently, DAC [ 13 ] introduced a multimodal dynamic division strategy that partitions noisy datasets based on the multimodal sample loss distribution. However, this method overlooks the interaction between different modality outputs, making it susceptible to the choice of division thresholds. To address these challenges, we propose a robust 2D-3D Multi-level Cross-modal Adaptive Correction and Alignment framework (MCA), which consists of Multimodal Joint label Correction (MJC) and Multi-level Adaptive Alignment (MAA). Specifically, MJC utilizes multimodal historical self-predictions to reliably refine noisy labels, allowing the model to extract valuable semantic information from the noisy data. With the corrected labels, different samples are dynamically processed using varying alignment strategies: center-level, group-level, and instance-level alignments. Through this adaptive alignment approach, MCA effectively captures both semantic and intrinsic geometric information across modalities, accommodating varying noise ratios and facilitating the learning of robust, discriminative, and modality-invariant representations. Our key contributions are summarized as follows: • We propose a novel and robust 2D-3D Multi-level cross-modal adaptive Correction and Alignment framework (MCA) that effectively corrects noisy labels and adaptively align the cross-modal features. • In MCA, we introduce Multimodal Joint Label Correction (MJC), which leverages multimodal information to reliably refine noisy labels and enhance feature quality. • In MCA, we propose a Multi-level Adaptive Alignment (MAA) strategy to effectively align cross-modal features across multiple levels while mitigating the adverse effects of label noise. • Our method exhibits remarkable effectiveness and robustness, achieving state-of-the-art performance across both traditional and real-world 3D object benchmarks. 2 Related Work 2.1 Cross-modal Retrieval Cross-modal retrieval methods aim to unify heterogeneous modalities within a shared representation space. Some approaches achieve this by leveraging statistical dependencies, such as Canonical Correlation Analysis (CCA) [ 14 ] and Multi-View Learning [ 15 ] . More recently, methods like contrastive learning [ 5 ] enhance alignment by maximizing mutual information between paired data. Other strategies rely on explicit supervision to enforce modality-invariant representations. For example, [ 12 ] optimizes both label and feature space alignment, while metric learning techniques [ 9 ] use center loss to refine intra- and inter-class relationships. To handle noisy annotations, [ 13 ] introduces a robust loss function. However, most existing methods assume clean data, limiting their effectiveness in 2D-3D retrieval with noisy labels. 2.2 Learning with Noisy Labels Learning with noisy labels is a critical challenge in machine learning, with existing methods typically falling into three categories: robust loss functions, label correction, and sample selection. Robust loss functions mitigate the impact of label noise by designing noise-tolerant objectives, reducing overfitting on corrupted labels. Label correction techniques refine noisy annotations through model-driven self-correction or external meta-correctors for higher-quality adjustments [ 16 ] . Sample selection approaches, on the other hand, aim to identify clean instances within noisy datasets. The widely used small-loss trick selects low-loss samples as reliable data points [ 17 , 18 ] , while Dividemix [ 8 ] leverages a Gaussian Mixture Model (GMM) to further distinguish clean and noisy instances. Additionally, historical model predictions offer an alternative selection criterion, enriching the decision process and improving reliability [ 19 , 4 ] . Recently, DAC [ 13 ] introduced a multimodal division strategy that dynamically separates clean and noisy samples by modeling the multimodal loss distribution. However, it overlooks the intrinsic relationships between different modalities. This limitation motivates us to develop a more reliable approach that not only distinguishes samples effectively but also refines noisy labels by deeply modeling cross-modal relationships. Figure 1: An overview of our proposed MCA framework. Our model first refines noisy labels using multimodal historical self-predictions through our MJC mechanism. It then adaptively enhances cross-modal feature semantics and discrimination through our MAA strategy. 3 Methodology 3.1 Overview Following the setting in [ 13 ] , we define the 2D-3D multimodal dataset as: \mathcal{D}=\left\{\mathcal{M}_{j}\right\}_{j=1}^{M}=\left\{\mathcal{X}_{j},\mathcal{Y}_{j}\right\}_{j=1}^{M} where M represents the number of modalities, and the dataset consists of K distinct categories. For each modality \mathcal{M}_{j} , the dataset contains N samples, formulated as: \mathcal{M}_{j}=\left\{\left(\boldsymbol{x}_{i}^{j},y_{i}^{j}\right)\right\}_{i=1}^{N} where \boldsymbol{x}_{i}^{j} denotes the i -th sample in the j -th modality, and y_{i}^{j} is its corresponding noisy label. For each sample x_{i}^{j} , we extract the sample feature z_{i}^{j}=\phi_{j}(x_{i}^{j}) with the corresponding feature extractor \phi_{j} . For simplicity, we denote (x_{j},y_{j})=(x_{i}^{j},y_{i}^{j}) in the following sections. The overview of our framework MCA is shown in Fig.

Read the complete text