Connecting Neural Models Latent Geometries with Relative Geodesic Representations
Hanlin YuUniversity of HelsinkiBerfin InalUniversity of AmsterdamGeorgios ArvanitidisDTUSøren HaubergDTUFrancesco LocatelloIST Austria, Marco FumeroIST Austria · 2025
Read Connecting Neural Models Latent Geometries with Relative Geodesic Representations by Hanlin YuUniversity of HelsinkiBerfin InalUniversity of AmsterdamGeorgios ArvanitidisDTUSøren HaubergDTUFrancesco LocatelloIST Austria, Marco FumeroIST Austria (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01599.txt.
Introduction
Hanlin Yu and five co-authors compare the hidden shapes that unlike neural networks learn for the same data, in a 2025 paper. Networks trained on the same tasks still build unlike inner maps, thanks to chance, design, and training biases. Yet shared base structure keeps the gaps between representations much the same, with some warp.
The authors go further using differential geometry. If two models trace roughly the same manifold, they build a map from the pullback metric, which captures the latent space's own shape. The method scales to large models. Tests on stitching pieces of unlike networks together, and on retrieval, span autoencoders and vision foundation models across designs and training schemes. The results back shape-aware checks as a useful way to align, reuse, and audit what models have learned.
Catalog record
| CREATOR | Hanlin YuUniversity of HelsinkiBerfin InalUniversity of AmsterdamGeorgios ArvanitidisDTUSøren HaubergDTUFrancesco LocatelloIST Austria, Marco FumeroIST Austria |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Connecting Neural Models Latent Geometries with Relative Geodesic Representations Hanlin YuUniversity of HelsinkiBerfin InalUniversity of AmsterdamGeorgios ArvanitidisDTUSøren HaubergDTUFrancesco LocatelloIST Austria, Marco FumeroIST Austria Abstract Neural models learn representations of high-dimensional data on low-dimensional manifolds. Multiple factors, including stochasticities in the training process, model architectures, and additional inductive biases, may induce different representations, even when learning the same task on the same data. However, it has recently been shown that when a latent structure is shared between distinct latent spaces, relative distances between representations can be preserved, up to distortions. Building on this idea, we demonstrate that exploiting the differential-geometric structure of latent spaces of neural models, it is possible to capture precisely the transformations between representational spaces trained on similar data distributions. Specifically, we assume that distinct neural models parametrize approximately the same underlying manifold, and introduce a representation based on the pullback metric that captures the intrinsic structure of the latent space, while scaling efficiently to large models. We validate experimentally our method on model stitching and retrieval tasks, covering autoencoders and vision foundation discriminative models, across diverse architectures, datasets, and pretraining schemes. 1 1 footnotetext: Corresponding emails: [email protected] , [email protected] 1 Introduction Figure 1 : Neural models trained on similar data learn parametrizations of the same manifold. NNs learn parametrizations ( D_{1},D_{2} ) of the same underlying manifold \mathcal{Y} up to isometric transformations \mathcal{T} . By pulling back the metric from \mathcal{Y} , relative geodesic representations are invariant to the transformation \mathcal{T} between latent spaces \mathcal{Z}_{1} and \mathcal{Z}_{2} Neural models learn meaningful representations of high-dimensional data generalizing to many tasks, spanning different data modalities and domains. Recent research reveals that these models often develop similar internal representations when exposed to similar inputs Li et al. 2015 ; Moschella et al. 2023 ; Fumero et al. 2024 ; Kornblith et al. 2019a , a phenomenon that was observed in biological networks Laakso and Cottrell 2000 ; Haxby et al. 2001 . Remarkably, even when models have different architectures, their internal representations can frequently be aligned through a simple, e.g., orthogonal, transformation Maiorca et al. 2024 ; Lähner and Moeller 2024a ; Moayeri et al. 2023 . This suggests a certain consistency in how neural nets encode information, emphasizing the importance of studying these internal representations, and the transformations that relate them, to the extent to hypothesize whether neural nets are converging toward a single representation of reality Huh et al. 2024 . One strategy to understand how different models are related is to identify representations that are invariant to transformations between distinct models’ representational spaces. A simple and effective recipe is that of relative representations Moschella et al. 2023 , where samples are represented as a function of a fixed set of latent representations. The similarity function employed is cosine similarity, hinting at the fact that representations across distinct models are subject to angle preserving transformations. However, the choice of similarity function should not be limited to only capturing invariances of one class of transformations. As shown in Cannistraci et al. 2024 ; Fumero et al. 2021 , other choices can be good as well, and there’s not a clear best choice among different transformations for capturing transformation across distinct latent spaces.We posit that when it is possible to relate distinct neural models’ representational spaces, this suggests that neural models are learning distinct parametrizations of the same underlying manifold (see Figure 1 ). In this paper, we employ geodesic distance in the latent space as a metric for relative representations. This approach ensures that the relative space remains approximately invariant to the isometries and reparametrization of the data’s manifold, as characterized by a Riemannian structure. Our contributions can be summarized as follows: • We observe that distinct neural models learn parametrization of the same underlying manifold when trained on similar data. • We propose a new representation that captures the isometric transformation between data manifolds learned by distinct models, by leveraging the pullback metric. • We propose to employ a scalable approximation of the geodesic energy to compute intrinsic distances that preserve the ranks of true distances. • We observe that different pullback metrics are suitable for different tasks, showing for the first time how to get meaningful pullback metrics from discriminative models, such as classifiers and self-supervised models. • We test relative geodesics on retrieval and stitching tasks on autoencoders and real vision foundation models, across different seeds, architectures, and training strategies, outperforming previous methods. 2 Related Work Representation alignment. Numerous studies have shown that neural networks trained under different initializations, architectures, or objectives learn highly similar internal feature representations Bonheme and Grzes 2022 ; Kornblith et al. 2019b ; Klabunde et al. 2023 ; Li et al. 2015 ; Bengio et al. 2014 ; Maiorca et al. 2024 ; Huh et al. 2024 ; Guth et al. 2024 ; Chang et al. 2022 ; Conneau et al. 2018 ; Tsitsulin et al. 2020 ; Nejatbakhsh et al. 2024 . This correspondence becomes stronger in wide and large networks Barannikov et al. 2022 ; Morcos et al. 2018 ; Somepalli et al. 2022 . Leveraging these aligned embeddings, a simple linear transformation often suffices to map one network’s latent space onto another’s, enabling techniques such as model stitching, where components from different networks can be interchanged with minimal loss in performance Fumero et al. 2024 ; Bansal et al. 2021 ; Csiszárik et al. 2021 . In practice, aligning two independently learned latent spaces often requires only a linear transformation, which achieves comparable downstream task performance ( Moayeri et al. 2023 ; Merullo et al. 2023 ; Maiorca et al. 2024 ; Lähner and Moeller 2024b ) . Latent space geometry. Early work on the geometry of deep latent representations focused on autoencoders, where the decoder’s mapping from latent to data space induces a natural pull-back metric under the assumption that the ambient space is Euclidean ( Shao et al. 2018 ; Tosi et al. 2014 ; Arvanitidis et al. 2018 ) . The Riemannian viewpoint allows one to compute geodesic paths and meaningful distances that respect the manifold structure of the learned embedding. Subsequent research has introduced computationally efficient approximations, such as energy-based proxies, and extended these ideas to estimate local curvature for improved interpolation and sampling ( Chen et al. 2019 ; Chadebec and Allassonnière 2022 ; Loaiza-Ganem et al. 2024 ; Arvanitidis et al. 2021 ; Arvanitidis et al. 2022a ) . In the context of discriminative models, one can obtain a Riemannian metric primarily using two approaches ( Grosse 2022 ) , either by pulling back the Fisher Information Matrix ( Amari 2016 ; Arvanitidis et al. 2022b ) or by assuming a Euclidean geometry on logit space and pulling back the metric. 3 Method 3.1 Notation and Background Neural networks (NNs) are parametric functions F_{\theta} , composed of an encoding map and a decoding map, represented as F_{\theta}=D_{\theta_{2}}\circ E_{\theta_{1}} . The encoder E_{\theta_{1}}:\mathcal{X}\mapsto\mathcal{Z} generates a latent representation z=E_{\theta_{1}}(x) , where x\in\mathcal{X} to the input domain \mathcal{X} , and the latent space \mathcal{Z} . The decoder D_{\theta_{2}} is responsible for performing the task at hand, such as reconstruction or classification. For simplicity, we omit the parameter dependence ( \theta ) in our notation moving