Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis
Dylan B. Lewis, Jens Gregor, Hector Santos-Villalobos · 2026
Read Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis by Dylan B. Lewis, Jens Gregor, Hector Santos-Villalobos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00921.txt.
Introduction
Lewis, Gregor, and Santos-Villalobos propose a simple training-free method to slim image vectors using a post-hoc canonical correlation analysis step. By using shared structure between two trained models, the method finds linear maps that keep shared meaning while dropping extra parts. Vectors shrink by more than 75 percent in size with better downstream results. They beat standard and PCA versions alike, with scores up to 12.6 percent.
Catalog record
| CREATOR | Dylan B. Lewis, Jens Gregor, Hector Santos-Villalobos |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33303 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00921
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis Dylan B. Lewis, Jens Gregor, Hector Santos-Villalobos Abstract Modern vision pipelines increasingly rely on pretrained image encoders whose representations are reused across tasks and models, yet these representations are often overcomplete and model-specific. We propose a simple, training-free method to improve the efficiency of image representations via a post-hoc canonical correlation analysis (CCA) operator. By leveraging the shared structure between representations produced by two pre-trained image encoders, our method finds linear projections that serve as a principled form of representation selection and dimensionality reduction, retaining shared semantic content while discarding redundant dimensions. Unlike standard dimensionality reduction techniques such as PCA, which operate on a single embedding space, our approach leverages cross-model agreement to guide representation distillation and refinement. The technique allows representations to be reduced by more than 75% in dimensionality with improved downstream performance, or enhanced at fixed dimensionality via post-hoc representation transfer from larger or fine-tuned models. Empirical results on ImageNet-1k, CIFAR-100, MNIST, and additional benchmarks show consistent improvements over both baseline and PCA-projected representations, with accuracy gains of up to 12.6%. I Introduction Modern vision systems increasingly rely on large pretrained encoders whose representations are reused across tasks, datasets, and downstream pipelines [ 11 , 5 , 15 , 2 ] . While these representations are highly expressive, they are often overcomplete and model-specific, and there is limited guidance on which dimensions capture task-relevant semantic structure across models as opposed to incidental or redundant variation. Common dimensionality reduction techniques such as principal component analysis (PCA) operate on single-model statistics [ 17 ] , making it difficult to distinguish generally useful structure from artifacts specific to a particular training objective or architecture. At the same time, it has become increasingly common to work with multiple pretrained models—often trained using different self-supervised objectives or fine-tuned to different extents—without retraining them jointly [ 26 , 19 ] . Prior work has explored representation alignment and comparison across networks primarily as an analysis tool rather than as a mechanism for modifying representations themselves [ 27 , 22 ] . This raises a natural question: can agreement between independently trained representations be leveraged directly to identify and reuse shared semantic structure post-hoc, without additional optimization or supervision Fig. 1: CCA cross-model alignment pipeline. (a) A backbone encoder M_{X} and a partner encoder M_{Y} produce baseline embeddings from the same input image. (b) CCA projects both embeddings into a shared, maximally correlated subspace. (c) The aligned space enables dimensionality reduction while transferring shared structure across models. (d) The resulting representations are evaluated on a downstream classification task. In this paper, we re-purpose canonical correlation analysis (CCA) as a tool for cross-model representation selection, identifying directions in two embedding spaces that are mutually predictable and therefore shared across independently trained encoders. This perspective re-frames CCA from an analysis technique into an operator for aligning and filtering representations based on cross-model agreement. We emphasize that our goal is not to compete directly with state-of-the-art methods specialized for distillation, dimensionality reduction, or representation learning, which typically rely on task-specific training objectives. Instead, we study a complementary setting in which representations are aligned and refined post-hoc, without retraining, supervision, or architectural modification. Figure 1 illustrates the data and process flow for our method. We show the effectiveness of our method on a variety of image classification datasets, including ImageNet-1k, CIFAR-100, and MNIST. The main findings of this work are the following: • Post-hoc representation selection via agreement: We repurpose canonical correlation analysis as a post-hoc representation selection operator, showing that cross-model agreement identifies semantically useful subspaces shared across independently trained image encoders M_{X} and M_{Y} . • Agreement-based selection outperforms variance-based reduction: We empirically demonstrate that CCA-based projections consistently improve linear evaluation performance relative to both unprojected baseline representations and PCA-based dimensionality reduction, indicating that cross-model agreement is a stronger inductive bias than single-model variance. • Post-hoc transfer of fine-tuning signal without training: We show that when two encoders share pre-training but only one is fine-tuned on a downstream task, CCA-based alignment enables post-hoc transfer of task-relevant structure, improving performance for both fine-tuned and non-fine-tuned models without additional optimization. • Deterministic, model-driven dimensionality selection: We demonstrate that CCA induces a deterministic target dimensionality based on model pair geometry, enabling substantial compression without dataset-specific tuning. • Robustness across datasets and identification of failure modes: We validate the method across multiple vision benchmarks and characterize class imbalance as a structured failure mode, providing guidance on when CCA-based alignment is reliable. This work provides a PyTorch [ 25 ] implementation of the CCA computation from [ 27 , 22 ] which makes the CCA computation faster by utilizing GPUs. II Related Work II-A Canonical Correlation Analysis Canonical correlation analysis (CCA) is a classical technique from multivariate statistics [ 18 ] that determines how to linearly project two datasets into a maximally correlated, joint-dimensional latent space by solving a pair of generalized eigenvalue problems. Generalizations include kernel CCA [ 3 , 13 ] , which finds nonlinear dependencies between datasets, and multi-set CCA [ 23 ] , which extends application to multiple datasets. CCA has been applied to signal processing [ 29 ] , unlabeled clustering [ 6 ] , multi-view learning [ 33 ] , and statistical data fusion [ 23 ] , serving as a foundational tool for understanding relationships between heterogeneous representations. Deep CCA [ 1 ] and related derivatives [ 4 ] , [ 35 ] learn nonlinear relationships between datasets using gradient-based optimization which enables richer cross-view correspondences. However, these methods more closely resemble traditional multi-view and contrastive representation learning methods. In addition, CCA has been widely used to analyze and compare internal representations learned by neural networks. Methods such as SVCCA [ 27 ] and projection-weighted CCA [ 22 ] employ CCA to quantify representational similarity across architectures, layers, or training stages, providing insight into how networks evolve during learning. In these works, CCA serves as a diagnostic tool: the learned projections are used to measure similarity but are not applied to modify or deploy representations in downstream tasks. In contrast, our work uses CCA-derived projections directly as representation transformations, enabling post-hoc alignment, selection, and reuse of representations without additional training. II-B Dimensionality Reduction Dimensionality reduction is a well-studied problem. Widely used techniques include principal component analysis (PCA) [ 17 ] and linear discriminant analysis (LDA) [ 12 ] , Non-linear variants of PCA [ 30 ] and LDA [ 28 ] also exist, but our work utilizes the classical, linear formulations of these methods. PCA based dimensionality reduction retains the dimensions that account for most of the variance within a single dataset, while LDA retains the dimensions that capture the most intra-class variance (directions that separate classes the best). In either case, the number of dimensions retained depends on intrinsic properties of the data considered. CCA, on the other hand, produces a space with a pre-defined dimensionality. We propose using this property to guide dimensionality reduction for image representations in a more deterministic manner. By using two models, one that yields a lower dimensional representation than the other, the high-dimensional representation can be compressed to the low-dimensional representation, regardless of the data itself. II-C Representation