Escaping Plato’s Cave: Towards the Alignment of 3D and Text Latent Spaces
Souhail Hadgi 1 Luca Moschella 2, Andrea Santilli 2 Diego Gomez 1 Qixing Huang 4 Emanuele Rodolà 2 Simone Melzi 3 Maks Ovsjanikov 1 1 École polytechnique 2 Sapienza University of Rome 3 University of Milano-Bicocca 4 The University of Texas at Austin · 2025
Read Escaping Plato’s Cave: Towards the Alignment of 3D and Text Latent Spaces by Souhail Hadgi 1 Luca Moschella 2, Andrea Santilli 2 Diego Gomez 1 Qixing Huang 4 Emanuele Rodolà 2 Simone Melzi 3 Maks Ovsjanikov 1 1 École polytechnique 2 Sapienza University of Rome 3 University of Milano-Bicocca 4 The University of Texas at Austin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05283.txt.
Introduction
Souhail Hadgi and seven colleagues present a 2025 computer vision paper on aligning the internal languages of AI models trained on different data. Models trained only on 3D shapes and models trained only on text each build feature spaces; recent work showed image and text spaces secretly resemble each other. Whether 3D joins that pattern was unknown.
Simply forcing the full spaces together after training worked poorly. The team's key move was to compare only well-chosen lower-dimensional slices of each space. Alignment quality jumped, improving matching and retrieval between shapes and descriptions. Analysis suggested the shared slices carry semantic meaning, while other parts keep pure geometry. The study sets a first baseline for connecting 3D and language understanding.
Catalog record
| CREATOR | Souhail Hadgi 1 Luca Moschella 2, Andrea Santilli 2 Diego Gomez 1 Qixing Huang 4 Emanuele Rodolà 2 Simone Melzi 3 Maks Ovsjanikov 1 1 École polytechnique 2 Sapienza University of Rome 3 University of Milano-Bicocca 4 The University of Texas at Austin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05283
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Escaping Plato’s Cave: Towards the Alignment of 3D and Text Latent Spaces Souhail Hadgi 1 Luca Moschella 2, Andrea Santilli 2 Diego Gomez 1 Qixing Huang 4 Emanuele Rodolà 2 Simone Melzi 3 Maks Ovsjanikov 1 1 École polytechnique 2 Sapienza University of Rome 3 University of Milano-Bicocca 4 The University of Texas at Austin Abstract Recent works have shown that, when trained at scale, uni-modal 2D vision and text encoders converge to learned features that share remarkable structural properties, despite arising from different representations. However, the role of 3D encoders with respect to other modalities remains unexplored. Furthermore, existing 3D foundation models that leverage large datasets are typically trained with explicit alignment objectives with respect to frozen encoders from other representations. In this work, we investigate the possibility of a posteriori alignment of representations obtained from uni-modal 3D encoders compared to text-based feature spaces. We show that naive post-training feature alignment of uni-modal text and 3D encoders results in limited performance. We then focus on extracting subspaces of the corresponding feature spaces and discover that by projecting learned representations onto well-chosen lower-dimensional subspaces the quality of alignment becomes significantly higher, leading to improved accuracy on matching and retrieval tasks. Our analysis further sheds light on the nature of these shared subspaces, which roughly separate between semantic and geometric data representations. Overall, ours is the first work that helps to establish a baseline for post-training alignment of 3D uni-modal and text feature spaces, and helps to highlight both the shared and unique properties of 3D data compared to other representations. Figure 1 : A visual overview of the proposed approach. is illustrated. From left to right: We begin with two distinct input collections—one consisting of 3D shapes and the other of textual prompts. In the blue box, independent, frozen uni-modal encoders map each modality into separate, high-dimensional latent spaces, shown in the red box. A dimensionality reduction procedure is applied to project these learned spaces into low-dimensional subspaces, represented in the green box. Finally, an alignment method registers the two low-dimensional subspaces, enabling cross-modal applications such as shape retrieval, with examples depicted in the yellow box. 1 Introduction Recent advances in multi-modal learning, particularly in vision-language models such as CLIP Radford et al. 2021 , have sparked interest in extending these successes to the 3D domain. Most current approaches primarily focus on training 3D encoders through triplet-based learning with pre-trained 2D vision and language encoders Liu et al. 2024 ; Xue et al. 2024a ; Zhou et al. 2023 , leveraging new large-scale datasets such as Objaverse Deitke et al. 2022 ; Deitke et al. 2023 , and showing promising results in tasks such as zero-shot shape recognition. While CLIP Radford et al. 2021 was trained with explicit alignment objectives between text and image representations, recent work has observed that even when trained independently , vision and text encoders tend to exhibit significant similarities Huh et al. 2024 ; Maniparambil et al. 2024 . In particular, the learned latent spaces of pure (uni-modal) text and vision encoders have similar proximity structure Huh et al. 2024 and can be aligned relatively easily after training with a small number of known anchor correspondences Maniparambil et al. 2024 ; Moschella et al. 2023 ; Cannistraci et al. 2023 . Furthermore, the degree of similarity in learned features across modalities strongly correlates with the quality of performance in various downstream tasks. One prominent interpretation of these results is that given sufficient scale of training data and model complexity, different representations are converging to a shared underlying structure of the physical world. This has given rise to the Platonic Representation Hypothesis Huh et al. 2024 , where different representations are viewed as projections of reality on particular modalities. At the same time, as the physical world is inherently (at least) 3-dimensional, a natural question is how the structure of uni-modal vision or text encoders relates to the features learned directly from 3D data. This question raises several challenges: first, large-scale 3D datasets have only recently become available Deitke et al. 2022 ; Deitke et al. 2023 ; second, most existing 3D “foundation models” are trained with explicit alignment objectives with respect to frozen 2D and text encoders, which limits the utility of post-training comparison. Finally, there is a lack of universally agreed-upon architectures and training objectives for 3D data, and many commonly-used architectures tend to have limited generalization power Xie et al. 2020 . In this work, we initiate the first study on the relation between 3D and language representations. We formulate the task of post-training alignment between a range of 3D and text encoders and study the accuracy and utility of several alignment strategies. Our first insight is that when trained on pure 3D data with self-supervised objectives, 3D encoders tend to lead to representations that align only very weakly to text-based representations. We believe that this observation already highlights the difficulty of uni-modal training and sheds light on the scarcity of “pure” 3D foundation models. Additionally, we reveal that while 3D and text latent spaces are not naturally aligned, effective cross-modal translation can be achieved through subspace projection and alignment . Our key insight is that by identifying and operating in correlated subspaces, we can improve the latent space alignment of 3D and text encoders without the need for expensive joint training. To achieve this, we introduce a simple but effective approach that combines Canonical Correlation Analysis (CCA) and existing alignment approaches such as affine translation Maiorca et al. 2024 and local CKA Maniparambil et al. 2024 . This approach and our subsequent analysis extend previous works aimed at comparing learned feature spaces, but shows that careful subspace alignment can reveal subtle but important similarities, which are otherwise obscured in global comparisons. We provide a visual overview of the proposed approach in Fig. 1 . To summarize, our main contributions are as follows: 1. We extend the analysis of vision-text uni-modal latent space alignment to 3D uni-modal encoders and text, highlighting the limited similarity between these latent spaces and the low efficacy of current alignment approaches. 2. We propose an efficient approach for cross-modal alignment between text and 3D features that combines CCA for subspace selection and existing alignment methods. This approach improves alignment performance with minimal computational overhead, as demonstrated through experiments in matching and retrieval tasks. Our method establishes a baseline for 3D-text cross-modal understanding, providing an alternative to explicit multimodal pre-training for cross-modal tasks. 3. We observe a complementary structure between the spanned subspaces and the original feature spaces, enabling a distinction between geometric and semantic representations. 2 Related Work Multi-modal representation learning. Multimodal representation learning has surged in recent years, driven by the success of image-language models Radford et al. 2021 ; Saharia et al. 2022 ; Li et al. 2022b ; Jia et al. 2021 that enable seamless cross-modal applications. These models serve as the backbone for tasks spanning from text-based image retrieval to generating high-quality visuals from natural language prompts. By aligning visual and linguistic features in a shared latent space, these models have paved the way for advanced interactions between modalities, setting the stage for applications where visual and textual information are jointly processed and understood.