Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion

Eghbal A. Hosseini, Brian Cheung, Evelina Fedorenko & Alex H. Williams · 2026

Read Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion by Eghbal A. Hosseini, Brian Cheung, Evelina Fedorenko & Alex H. Williams (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21836.txt.

Introduction

Hosseini and three colleagues measure how single stimuli drive convergence between neural networks. Using a Generalized Procrustes method, they score intra-modal agreement among vision models with different training goals. Stimuli that vision models agree on most also align far better with language models, by up to a factor of two in DINOv2 pairings. The effect holds across selection criteria and model pairings, suggesting stimulus-level agreement shapes cross-modal convergence.

Catalog record

CREATOREghbal A. Hosseini, Brian Cheung, Evelina Fedorenko & Alex H. Williams
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16837
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21836

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion Eghbal A. Hosseini, Brian Cheung, Evelina Fedorenko & Alex H. Williams Abstract Neural networks exhibit a remarkable degree of representational convergence across diverse architectures, training objectives, and even data modalities. This convergence is predictive of alignment with brain representation. A recent hypothesis suggests this arises from learning the underlying structure in the environment in similar ways. However, it is unclear how individual stimuli elicit convergent representations across networks. An image can be perceived in multiple ways and expressed differently using words. Here, we introduce a methodology based on the Generalized Procrustes Algorithm to measure intra-modal representational convergence at the single-stimulus level. We applied this to vision models with distinct training objectives, selecting stimuli based on their degree of alignment (intra-modal dispersion). Crucially, we found that this intra-modal dispersion strongly modulates alignment between vision and language models (cross-modal convergence). Specifically, stimuli with low intra-modal dispersion (high agreement among vision models) elicited significantly higher cross-modal alignment than those with high dispersion, by up to a factor of two (e.g., in pairings of DINOv2 with language models). This effect was robust to stimulus selection criteria and generalized across different pairings of vision and language models. Measuring convergence at the single-stimulus level provides a path toward understanding the sources of convergence and divergence across modalities, and between neural networks and human neural representations. 1 Introduction Artificial neural networks show a remarkable ability to learn representations that generalize across tasks, and predict neural representations in humans and other animals. Even though details of implementation — including training, architecture, and input modality — vary between networks, they appear to have minimal influence on the final representation, and models converge onto similar representations. The Platonic Representation Hypothesis ( 9 ) suggests this convergence arises from learning shared environmental priors. 8 found similar evidence in convergence across neural networks and their predictivity of brain representation. Both lines of work however established convergences across groups of stimuli, and thus globally. Individual observations are critical for probing and understanding representations and can potentially drive models, and humans, toward either convergence or divergence. Humans often find themselves interpreting differently the same work of art, a painting for example. Existing methods are often ill-suited for probing convergence at single stimulus level, as they either measure local effects ( 2 ) , average similarity over a large set of stimuli datasets ( 8 ) , or lack a proper metric space for rigorous comparison ( 13 ; 5 ) . This highlights the need for a robust, stimulus-specific measure of alignment. Building on Generalized Procrustes Analysis ( 4 ) , we introduce a stimulus-specific measure of representational convergence across vision models ( 15 ; 6 ) . We then demonstrate that our measure can be used to select stimuli that effectively modulate representational alignment across modalities, from vision to language. 2 Methods 2.1 Generalized Procrustes method Generalized Procrustes Analysis (GPA) is a method for aligning multiple datasets, or embedding spaces, into a common reference frame. The core idea is to find a set of optimal transformations, restricted to orthogonal transformations, that minimize the discrepancy between a shared consensus configuration and each individually transformed dataset ( 4 ) . Stimulus-level residuals are a classical by-product of GPA; our contribution is their application to comparing neural network representations and to modulating cross-modal alignment. Formally, consider a set of M representation matrices \{N_{1},N_{2},\dots,N_{M}\} , where each representation N_{i}\in\mathbb{R}^{m\times n_{i}} , with i\in\{1,\dots,M\} indexing models, represents m corresponding samples (e.g., stimuli) in an n_{i} -dimensional space. The dimensionality n_{i} can differ across the representations. GPA seeks to find an optimal consensus representation, N_{\text{joint}}\in\mathbb{R}^{m\times k} , and a set of corresponding orthogonal transformation matrices, \{T_{1},T_{2},\dots,T_{M}\} , where each T_{i}\in\mathbb{R}^{n_{i}\times k} , by minimizing the sum of squared Frobenius distances. The optimization problem is defined as: \min_{N_{\text{joint}},\{T_{i}\}_{i=1}^{M}}\sum_{i=1}^{M}\|N_{i}T_{i}-N_{\text{joint}}\|_{F}^{2} (1) subject to each transformation matrix being orthogonal, T_{i}^{T}T_{i}=I_{k} for all i\in\{1,\dots,M\} . Here, \|\cdot\|_{F} denotes the Frobenius norm, and the constraint T_{i}^{T}T_{i}=I_{k} ensures that the transformations do not distort the internal geometry of each representation N_{i} . This process effectively rotates each embedding space to achieve maximal alignment with the emergent consensus space N_{\text{joint}} . Prior to performing GPA, we first center each embedding along the second dimension n_{i} , zero-pad dimensions across all models to a common dimensionality, and normalize each representation \|N_{i}\|_{F}=1 . Generalized Procrustes Analysis can also be interpreted as computing a barycenter of neural representations in Procrustes shape space. In classical shape analysis, each centered, scale-normalized configuration N_{i} corresponds to a point on a quotient manifold obtained by modding out rotations, 1 1 1 Here we include reflections as well as rotations in the equivalence class. This is common in the analysis of neural representations since even a permutation of the neural unit labels can result in a reflection. and Generalized Procrustes iteratively aligns all configurations to the barycenter (also called the Fréchet mean or Karcher mean), represented by N_{\text{joint}} . This perspective clarifies that GPA is not merely an arbitrary linear alignment, but a principled procedure for finding the central tendency of a population of neural representations with respect to a scale and rotation-invariant metric space ( 15 ) . We restrict each T_{i} to orthogonal transformations, rather than a general linear map, because they preserve the internal geometry of each representation: pairwise distances and angles between stimuli are unchanged under T_{i} . A general linear map can absorb scale and reshape covariance structure, which are not desirable for defining metrics on representation ( 15 ) . Scale is instead handled explicitly by the Frobenius normalization \|N_{i}\|_{F}=1 . 2.2 Quantifying Single-Stimulus Dispersion To quantify representational convergence at the single-stimulus level, we measure the dispersion between each model’s transformed representation and the shared joint representation, N_{\text{joint}} . Specifically, after computing the joint space via GPA, we first project each model’s representation into this common space using its corresponding transformation matrix, T_{i} . For each stimulus j\in\{1,\dots,m\} , we then compute the Euclidean distance between its projected representation from model i and its joint representation. This yields the Procrustes residual , r_{ij} , defined as: r_{ij}=\|(N_{i}T_{i})_{j}-(N_{\text{joint}})_{j}\|_{2} (2) This process results in a residual matrix R\in\mathbb{R}^{m\times M} , where each entry r_{ij} quantifies the dispersion for a given stimulus j and model i (see Fig 1 C). A low residual value signifies high alignment (low dispersion), while a high value indicates low alignment (high dispersion). We employ two strategies to summarize the overall dispersion for each stimulus across all M models: 1. Mean Dispersion: Our first approach is to average the residuals across all models for each stimulus. We rank the stimuli from least to most dispersed. While straightforward, this metric can be sensitive to outlier models (Fig 1 D) 2. Principal Component of Dispersion: To capture the primary axis of disagreement in a more robust manner, our second approach utilizes Principal Component Analysis (PCA) on the residual matrix R . We use the score along the first principal component (PC1) as a more nuanced measure of dispersion for each stimulus, as it reflects the most significant shared variance in model disagreement (Fig 1 E). 2.3 Vision Models We inspected representations across ViT models trained on diverse objectives ( 9 ) .

Read the complete text