Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations
Yujia Zhang Xiaoyang Wu Yixing Lao Chengyao Wang, Zhuotao Tian Naiyan Wang Hengshuang Zhao, project lead corresponding author https://pointcept.github.io/Concerto · 2025
Read Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations by Yujia Zhang Xiaoyang Wu Yixing Lao Chengyao Wang, Zhuotao Tian Naiyan Wang Hengshuang Zhao, project lead corresponding author https://pointcept.github.io/Concerto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23607.txt.
Introduction
Yujia Zhang and six co-authors introduce Concerto, a learning scheme that teaches machines spatial understanding by mixing 2D images and 3D point clouds. People form concepts by combining senses, then can recall them from a single sense alone. Concerto copies that idea in minimal form, pairing self-teaching inside the 3D data with a joint embedding that ties 2D and 3D views together.
Even with simple probes, the learned features beat leading standalone 2D and 3D self-supervised models on 3D scene perception, and also beat just gluing the two feature sets together. With full fine-tuning, Concerto set new best results on several scene understanding benchmarks, including a top score on ScanNet. The team adds a variant for point clouds lifted from video, plus a translator that maps its features into the CLIP language space. That last step opens the door to open-world perception driven by words.
Catalog record
| CREATOR | Yujia Zhang Xiaoyang Wu Yixing Lao Chengyao Wang, Zhuotao Tian Naiyan Wang Hengshuang Zhao, project lead corresponding author https://pointcept.github.io/Concerto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23607
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations Yujia Zhang Xiaoyang Wu Yixing Lao Chengyao Wang, Zhuotao Tian Naiyan Wang Hengshuang Zhao, project lead corresponding author https://pointcept.github.io/Concerto Abstract Humans learn abstract concepts through multisensory synergy, and once formed, such representations can often be recalled from a single modality. Inspired by this principle, we introduce Concerto, a minimalist simulation of human concept learning for spatial cognition, combining 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding. Despite its simplicity, Concerto learns more coherent and informative spatial features, as demonstrated by zero-shot visualizations. It outperforms both standalone SOTA 2D and 3D self-supervised models by 14.2% and 4.8%, respectively, as well as their feature concatenation, in linear probing for 3D scene perception. With full fine-tuning, Concerto sets new SOTA results across multiple scene understanding benchmarks (e.g., 80.7% mIoU on ScanNet). We further present a variant of Concerto tailored for video-lifted point cloud spatial understanding, and a translator that linearly projects Concerto representations into CLIP’s language space, enabling open-world perception. These results highlight that Concerto emerges spatial representations with superior fine-grained geometric and semantic consistency. Figure 1 : We visualize the principal components of point features learned by 2D and 3D self-supervised models [ 28 , 49 ] , mapped to RGB colors. DINOv2 lacks geometric awareness, and Sonata struggles to capture fine textures. Concerto integrates intra-modal self-distillation with cross-modal joint embedding prediction, enabling a self-supervised point cloud transformer [ 47 ] to learn richer, emerging spatial representations with fine-grained geometric and semantic consistency across views. 1 Introduction Learning strong spatial representations in a self-supervised manner is foundational for spatial cognition tasks, spanning from low-level machine perception to high-level reasoning in domains such as autonomous driving [ 38 , 8 ] , mixed reality [ 7 , 23 , 37 ] , and robotics [ 17 ] . Recent advances in self-supervised learning have significantly improved foundational representation models for the two dominant spatial modalities: 2D images [ 12 , 18 , 56 , 4 , 51 ] and 3D point clouds [ 50 , 29 , 53 , 46 , 49 ] . Without the need for human annotations, these models have demonstrated strong performance across various downstream tasks by enabling the learning of geometry, and semantics at scale. However, despite their individual successes within each data modality, our pilot study reveals that self-supervised representations learned independently from images and point clouds do not fully overlap. Specifically, concatenating features from self-supervised image models (e.g., DINOv2 [ 28 ] ) and point cloud models (e.g., Sonata [ 49 ] ) leads to improved linear probing performance, suggesting that each modality captures complementary, rather than redundant, aspects of spatial information. The observation hints at the existence of a more robust and rich feature space that emerges from the interaction between 2D and 3D modalities , indicating the core aim of this research: to uncover superior spatial representations through multi-modal self-supervised learning . Figure 2 : The “Apple” concept in cognition. Our inspiration toward this target is rooted in how humans learn abstract concepts: through multisensory synergy [ 6 , 36 ] . Consider the example of an apple (as illustrated in Fig. 2 )—our understanding of it is not limited to its visual appearance. Instead, the concept is formed through repeatedly seeing, touching, and tasting apples, allowing us to internalize its geometry, texture, and semantic meaning in a unified, predictive way. This cognitive process reflects a continuous, cross-modal integration of sensory data into a grounded concept space (Fig. 2 top). Yet once such a representation is formed, it can be evoked from just a single modality: seeing an image of an apple can vividly recall its weight and texture, just as holding one can bring its color and shape to mind (Fig. 2 bottom). This ability to retrieve rich, structured knowledge from partial sensory input underscores the importance of learning modality-agnostic representations that are both unified and predictive. Driven by this vision, our methodology aims to offer a simple yet effective imitation of human multisensory synergy. To this end, we compose a Concerto of 2D-3D joint self-supervised learning, coupling intra-modal self-distillation for point representations [ 28 , 49 ] with cross-modal joint embedding prediction from images to point clouds [ 26 ] . Ultimately, this training yields a self-supervised PTv3 [ 47 ] model, pretrained on 40k raw point clouds and 300k images [ 15 , 52 , 1 , 7 , 33 , 60 , 5 ] . In addition, we present a variation of Concerto, augmented with an additional set of 50k point clouds with 200k corresponding images lifted from scene videos [ 61 ] via feed-forward reconstruction [ 43 ] , tailored for video-based spatial understanding. We also introduce an interlude of Concerto: a learned translator that linearly projects self-supervised representations into CLIP’s language space [ 32 , 27 ] , enabling open-world perception. The intersection of 2D and 3D self-supervised learning in Concerto yields a powerful synergy, enabling the emergence of superior spatial representations. PCA-colored visualizations reveal that Concerto captures more coherent and informative spatial features than SOTA 2D or 3D self-supervised models trained on a single modality (see Fig. 1 ). Concerto exceeds its predecessor, Sonata, with a 4.8% improvement in linear probing, achieving 77.3% mIoU on ScanNet semantic segmentation using a single linear layer. Notably, this performance also surpasses the concatenation of Sonata and DINOv2 (1.4%), demonstrating that the multisensory synergy in Concerto exceeds the representational upper bound achievable by single-modality self-supervised learning. With full fine-tuning, Concerto achieves SOTA performance across a range of scene perception tasks. For example, reaching 80.7% mIoU on ScanNet semantic segmentation. Semantic Segmentation ScanNet Val [ 15 ] ScanNet200 Val [ 35 ] Method Type Encoder mIoU mAcc allAcc mIoU mAcc allAcc DINOv2 [ 28 ] 2D Image SSL ViT-G 63.09 75.50 82.42 27.42 37.59 72.80 Sonata [ 49 ] 3D Point SSL PTv3-B 72.52 83.11 89.74 29.25 41.61 81.15 Sonata \times DINOv2 3D SSL \times 2D SSL Both 75.91 85.36 91.25 36.67 46.98 82.85 Concerto (ours) 2D-3D Joint SSL PTv3-B 77.32 86.58 91.74 37.41 49.49 83.29 Table 1 : Linear probing results on 3D semantic segmentation. We compare self-supervised features learned from 2D, 3D, their feature concatenation, and our 2D-3D joint SSL model, Concerto (as a preview). Notably, the concatenation of 2D and 3D features outperforms either modality alone, suggesting that the two modalities encode complementary information. Concerto achieves the best performance across all metrics, demonstrating its ability to learn superior spatial representations. 2 Beyond Single Modality: Toward a New World of Representations This section presents a pilot study to explore high-level questions surrounding self-supervised representations. These questions form the conceptual foundation of our research, and our methodology emerges as a natural and simple response to the insights gained here. 2.1 Is There a Superior Representation Space Beyond Single-Modality Learning? Self-supervised learning on 2D images and 3D point clouds has achieved remarkable progress in visual representation learning. However, when trained independently, these models may capture only modality-specific perspectives of the spatial world. Just as a person who has only seen an apple but never tasted one may lack a sense of its flavor or texture, single-modal learning inevitably misses critical dimensions of the world.