Multi-Way Representation Alignment Akshit Achara King’s College London akshit.achara@kcl.ac.uk Tatiana Gaintseva Queen Mary University of London t.gaintseva@qmul.ac.uk, Mateo Mahaut Universitat Pompeu Fabra mateo.mahaut@gmail.com Pritish Chakraborty Indian Institute of Technology, Bombay pritish@cse.iitb.ac.in, Viktor Stenby Johansson Technical University of Denmark vstenby@dtu.dk Melih Barsbey Imperial College London m.barsbey@imperial.ac.uk, Emanuele Rodolà Sapienza University of Rome Paradigma rodola@di.uniroma1.it Donato Crisostomi Sapienza University of Rome crisostomi@di.uniroma1.it Abstract The Platonic Representation Hypothesis suggests that independently trained neural networks converge to increasingly similar latent spaces. However, current strategies for mapping these representations are inherently pairwise, scaling quadratically with the number of models and failing to yield a consistent global reference. In this paper, we study the alignment of M ≥ 3 M\geq 3 models. We first adapt Generalized Procrustes Analysis (GPA) to construct a shared orthogonal universe that preserves the internal geometry essential for tasks like model stitching. We then show that strict isometric alignment is suboptimal for retrieval, where agreement-maximizing methods like Canonical Correlation Analysis (CCA) typically prevail. To bridge this gap, we finally propose Geometry-Corrected Procrustes Alignment (GCPA), which establishes a robust GPA-based universe followed by a post-hoc correction for directional mismatch. Extensive experiments demonstrate that GCPA consistently improves any-to-any retrieval while retaining a practical shared reference space. 1 Introduction \Omega_{Y\leftarrow X} \Omega_{Z\leftarrow Y} \Omega_{X\leftarrow Z} X Y Z \Omega_{X}^{\top} \Omega_{Y}^{\top} \Omega_{Z}^{\top} \Omega_{X} \Omega_{Y} \Omega_{Z} X Y Z U Figure 1: Pairwise alignment (left) learns a separate map for each ordered pair, which does not enforce consistency when maps are composed. Universe alignment (right) learns one map per model into a shared reference U , enabling translation between models by composition. Deep networks succeed largely because of the representation spaces they build. Strikingly, these spaces often resemble each other even when the models differ in architecture, data, optimization, or random seed ( 26 ; 31 ; 30 ; 20 ) . This observation is captured by the Platonic Representation Hypothesis: independently trained models tend to recover a shared statistical structure of the world in their latent representations ( 20 ) . Motivated by these findings, a growing body of work studies representation alignment as a way to make independently trained models interoperable. Methods range from simple linear or orthogonal mappings ( 30 ) to richer functional correspondences ( 10 ) . When alignment succeeds, it unlocks practical use cases including model stitching, cross-modal transfer, and zero-shot composition across models ( 30 ; 31 ; 32 ) . Yet most alignment pipelines are still pairwise ( Figure 1 , left): they fit one map for each ordered pair of models and train each map independently. This is poorly matched to settings where many models are used together and translations are obtained by composing maps. In the M -model regime, pairwise alignment scales quadratically in the number of maps, requires fitting (M{-}1) new maps to incorporate a new model, and provides no mechanism to ensure that compositions are consistent. As a result, the translation from X to Z can differ depending on whether it is learned directly or obtained by composing through intermediate models, even when each pairwise map performs well under pairwise evaluation. This can make pairwise alignment unreliable as a shared reference for larger collections of models. A natural alternative is to factor translation through a shared universe space ( Figure 1 ), learning one map per model to and from a common reference. In the orthogonal case this yields a generalized Procrustes formulation. Translation between arbitrary model pairs is obtained via the universe, which reduces the number of learned maps to O(M) and provides a shared coordinate system that can be reused across tasks. Importantly, this construction encourages consistency under composition: the translation between any two models is unique and independent of the path taken through intermediate models. This supports multi-model use, simplifies model addition, and enables shared-coordinate analyses such as probing and aggregation. In this work, we study the alignment of three or more models through a shared universe reference. We first adapt Generalized Procrustes Analysis (GPA) ( 36 ; 12 ) to the representation alignment setting, demonstrating that its orthogonal nature yields a universe that preserves each model’s internal structure, a property essential for tasks requiring geometric fidelity such as model stitching. We then empirically show that geometry preservation alone is insufficient for tasks like zero-shot retrieval, as methods maximizing cross-model agreement such as Generalized Canonical Correlation Analysis (GCCA) ( 17 ; 24 ) outperform strict isometries. To bridge this gap, we propose Geometry-Corrected Procrustes Alignment (GCPA), which combines the best of both worlds. GCPA first establishes a robust orthogonal universe via GPA, then applies a post-hoc shared correction to minimize residual directional mismatch. We evaluate these approaches on benchmarks spanning inter-architecture probing, multilingual translation, and cross-camera retrieval. Our results show that while naturally multi-way techniques are necessary for scaling, the specific choice of objective matters: while GPA provides a consistent geometric reference, the proposed geometry-corrected universe (GCPA) is required to achieve state-of-the-art performance in any-to-any retrieval while retaining the practical benefits of the shared Procrustes frame. Contributions. 1. We extend orthogonal representation alignment to three or more models by adapting Generalized Procrustes Analysis (GPA) to the neural representation setting. 2. We demonstrate that while GPA provides a consistent global reference superior to pairwise maps, it lags behind agreement-maximizing methods (like GCCA) on retrieval tasks. 3. We introduce Geometry-Corrected Procrustes Alignment (GCPA) to bridge this gap, establishing a robust universe via GPA and applying a post-hoc correction for directional mismatch to achieve superior performance across diverse datasets and settings. 2 Related Work Representational similarity Work on representational similarity typically (i) measures how closely different learned spaces correspond ( 26 ; 25 ; 11 ; 20 ; 1 ) , (ii) leverages such similarity to align spaces via explicit maps ( 31 ; 30 ; 10 ; 2 ) , or (iii) studies similarity and alignment jointly ( 10 ) . The applications span multi-modal alignment ( 32 ; 5 ; 39 ; 14 ) , cross-view alignment ( 19 ) , as well as cross-lingual alignment ( 22 ) . While there are methods not requiring paired data ( 13 ; 23 ; 35 ) , we study in this paper the common case where the correspondence is available. A common and effective alignment tool in this line is Procrustes analysis, which fits an orthogonal map between two spaces and preserves their internal distances and angles ( 21 ; 13 ; 30 ) . More broadly, our setting can be viewed as a special case of multi-space representation learning focused on alignment ( 28 ) , and is connected to backward-compatible representation learning, which aims to keep updated embeddings comparable to earlier ones ( 37 ; 40 ) , as well as to communication between pretrained networks ( 29 ; 34 ) . Multiset alignment An alternative to pairwise mapping is to learn a shared latent space from multiple views via multiset canonical correlation analysis (GCCA) and related variants ( 17 ; 24 ; 9 ; 38 ) . These methods provide a principled M -way objective and are natural baselines for multi-view alignment, especially in retrieval-style settings where matched items should be directly comparable in a common set of coordinates. In contrast, our focus is on a single shared reference learned with one map per model, so that translations between models are obtained by composing through the reference and new models can be incorporated by fitting only one additional map. 3 Methodology We consider the problem of aligning M\geq 3 distinct representation spaces \{X_{m}\}_{m=1}^{M} , where X_{m}\in\mathbb{R}^{N\times d} contains N matched samples (or is padded to a common dimension). In our setup, all representations are standardized within each space. Our goal is to establish a shared coordinate system that enables any-to-any translation. 3.1 The Universe Factorization Standard alignment approaches are inherently pairwise. To map space n to space m , one typically minimizes the discrepancy between matched pairs: \mathcal{L}_{\text{pair}}(\Omega)=\sum_{m