Transformed Multi-view 3D Shape Features with Contrastive Learning

Márcus Vinícius Lobo Costa 1 , Sherlon Almeida da Silva 1 , Bárbara Caroline Benato 1 , Leo Sampaio Ferraz Ribeiro 1 , and Moacir Antonelli Ponti 1 1 Instituto de Ciências Matemáticas e de Computação (ICMC), Universidade de São Paulo (USP), São Carlos, SP, Brazil marcusvlc, [email protected], barbarabe · 2025

Read Transformed Multi-view 3D Shape Features with Contrastive Learning by Márcus Vinícius Lobo Costa 1 , Sherlon Almeida da Silva 1 , Bárbara Caroline Benato 1 , Leo Sampaio Ferraz Ribeiro 1 , and Moacir Antonelli Ponti 1 1 Instituto de Ciências Matemáticas e de Computação (ICMC), Universidade de São Paulo (USP), São Carlos, SP, Brazil marcusvlc, [email protected], barbarabe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19955.txt.

Introduction

Márcus Vinícius Lobo Costa and four co-authors test how well vision transformers, trained with contrast-based objectives, learn features of 3D shapes from many 2D views. Recognizing 3D objects from images usually demands wide labeled data. Convolutional networks can miss global shape relationships. The study pairs state-of-the-art backbones with supervised and self-supervised contrastive losses.

On downstream multi-view tasks, the transformer-plus-contrastive pairing performed strongly. A supervised contrast-based model reached about ninety-one percent accuracy on the ModelNet10 benchmark. The authors credit the transformer's grasp of overall shape. Contrastive training then refines that grasp into sharper local distinctions. Because the work is empirical, it rests on wide evaluation rather than theory to make its case.

Catalog record

CREATORMárcus Vinícius Lobo Costa 1 , Sherlon Almeida da Silva 1 , Bárbara Caroline Benato 1 , Leo Sampaio Ferraz Ribeiro 1 , and Moacir Antonelli Ponti 1 1 Instituto de Ciências Matemáticas e de Computação (ICMC), Universidade de São Paulo (USP), São Carlos, SP, Brazil marcusvlc, [email protected], barbarabe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transformed Multi-view 3D Shape Features with Contrastive Learning Márcus Vinícius Lobo Costa 1 , Sherlon Almeida da Silva 1 , Bárbara Caroline Benato 1 , Leo Sampaio Ferraz Ribeiro 1 , and Moacir Antonelli Ponti 1 1 Instituto de Ciências Matemáticas e de Computação (ICMC), Universidade de São Paulo (USP), São Carlos, SP, Brazil marcusvlc, [email protected], [email protected], leo.ribeiro, [email protected] Abstract This paper addresses the challenges in representation learning of 3D shape features by investigating state-of-the-art backbones paired with both contrastive supervised and self-supervised learning objectives. Computer vision methods struggle with recognizing 3D objects from 2D images, often requiring extensive labeled data and relying on Convolutional Neural Networks (CNNs) that may overlook crucial shape relationships. Our work demonstrates that Vision Transformers (ViTs) based architectures, when paired with modern contrastive objectives, achieve promising results in multi-view 3D analysis on our downstream tasks, unifying contrastive and 3D shape understanding pipelines. For example, supervised contrastive losses reached about 90.6% accuracy on ModelNet10. The use of ViTs and contrastive learning, leveraging ViTs’ ability to understand overall shapes and contrastive learning’s effectiveness, overcomes the need for extensive labeled data and the limitations of CNNs in capturing crucial shape relationships. The success stems from capturing global shape semantics via ViTs and refining local discriminative features through contrastive optimization. Importantly, our approach is empirical, as it is grounded on extensive experimental evaluation to validate the effectiveness of combining ViTs with contrastive objectives for 3D representation learning. The code is available at: https://github.com/usmarcv/RepLearningLosses . 1 INTRODUCTION Understanding 3D shapes is crucial for applications such as robotics and virtual reality, yet it remains a challenging task for computer vision systems. While humans perceive 3D structure from 2D views, computational models still face significant limitations. Current methods often rely on multiple 2D projections of 3D objects processed by Convolutional Neural Network (CNN) backbones trained under supervised learning. The representative work, Multi-View Convolutional Neural Network (MVCNN) [ Su et al., 2015 ] , made the first attempt at 3D shape understanding using CNN backbones. MVCNN was followed by further in-depth studies [ Hamdi et al., 2021 , Esteves et al., 2019 ] . However, these methods not only demand large labeled datasets but also struggle to capture complex shape relationships. In contrast, recent advances demonstrate that Vision Image Transformer (ViT) architectures frequently outperform CNNs in visual recognition tasks [ Caron et al., 2021 , Balestriero et al., 2023 ] , highlighting their potential for 3D understanding in multi-view learning settings. In a scenario of crescent interest, self-supervised learning has emerged as a powerful paradigm for leveraging unlabeled data [ Balestriero et al., 2023 ] . By designing pretext tasks that generate pseudo-labels directly from the input, self-supervised learning showed to be more robust while mitigates the heavy reliance on manual annotation [ Cavallari and Ponti, 2022 ] , which remains a major bottleneck for supervised approaches [ Dos Santos et al., 2020 ] . While supervised learning continues to play a central role in many downstream applications, self-supervised learning offers a scalable alternative for data-efficient 3D representation learning. In recent years, there has been a resurgence of works utilizing contrastive learning in many fields, particularly in settings with unlabeled examples. Contrastive learning operates in settings with both supervised and self-supervised learning with a central idea: “ pull together an anchor and a positive sample in an embedding space, and push apart the anchor from many negative samples ” [ Oord et al., 2018 , Khosla et al., 2020 ] . These works employ InfoNCE [ Oord et al., 2018 ] and SimCLR [ Chen et al., 2020 ] loss functions in supervised contrastive settings, such as SupCon [ Khosla et al., 2020 ] loss. Taking advantage of these approaches, some progress has been made in 3D shape understanding through contrastive learning in self-supervised settings [ Li et al., 2025 , Peng et al., 2024 ] . CMVL-Net [ Peng et al., 2024 ] utilizes embeddings generated by a CNN that relies on contrastive clustering via graph methods, while MVCL [ Li et al., 2025 ] uses a discriminative multi-view grouping that learns correlation between them using contrastive learning through a CNN backbone. However, contrastive learning settings have been considered only when combined with CNN backbones for 3D multi-view shape understanding. In this paper, we investigate how to integrate and leverage contrastive learning and ViT for 3D shape understanding with multi-view rendering . We propose to address the abovementioned limitations by exploring in particular four ViTs backbones and five contrastive learning losses, including self-supervised and supervised settings. We compare our proposed experiments with approaches using CNN backbones to evaluate what works best for 3D shape understanding in an end-to-end pipeline. We conduct our experiments in a way to answer the following research questions (RQ): RQ1. How CNN and ViT backbones perform for multi-viewing 3D shape? Following the findings in [ Chen et al., 2020 , Khosla et al., 2020 , Li et al., 2025 ] showing that ResNet-50 [ He et al., 2016 ] show beneficial performance in the classification task. In addition, we explored the ViT variants as ViT-S/16 [ Dosovitskiy et al., 2020 ] , ViT-B/16, DINO-S/16 [ Caron et al., 2021 ] and DINO-B/16. RQ2. How contrastive learning losses can enhance (or contribute to) multi-view 3D rendering pipelines? Besides the cross-entropy loss, we explored five alternatives to contrastive learning losses, such as InfoNCE [ Oord et al., 2018 ] , SimCLR [ Chen et al., 2020 ] , SupCon [ Khosla et al., 2020 ] , \epsilon -SupInfoNCE [ Barbano et al., 2023 ] , and SINCERE [ Feeney and Hughes, 2023 ] . RQ3. How combining contrastive learning losses with ViT for multi-view 3D shape can improve state-of-art methods? With these research questions, our experimental setup can be summarized by the model variants obtained from the combination of the choices listed in response to the research questions. Our set of contributions is then multi-faceted: • we show ViT backbones overcome CNN ones with contrastive learning losses for 3D multi-view images, an underexplored combination for representation learning in 3D shape understanding. • we present a comprehensive evaluation unifying the contrastive learning and 3D shape understanding pipelines. • in-depth analysis of feature learning breadth of side-by-side comparisons, with visualization, analysis of limitations of the learned embeddings, and generalization assessment with evaluation on image retrieval tasks. Outline. The paper is organized as follows. In Section 2 , we review deep learning, supervised and self-supervised learning, and contrastive learning under the multi-view 3D data. In Section 3 , we present our proposed approach that unifies the contrastive learning and 3D shape understanding pipelines used in this work. Then, in Section 4 we present the experimental protocol. In Section 5 , we show the results obtained for our downstream tasks. Finally, in Section 6 , we point out the contributions, limitations, and future work. 2 RELATED WORKS CNNs on 3D Multi-view data. Deep learning methods for 3D object representation typically fall into three categories: voxel-based, point-based, and view-based approaches [ Qi et al., 2021 ] . Looking to only view-based projections, the pioneering Multi-view Convolutional Neural Network (MVCNN) [ Su et al., 2015 ] introduced view-based methods with a unified CNN architecture, employing view

Read the complete text