The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?

Zhaoyang Zhang Run Shao Dongyue Wu Jiajie Teng, Chao Tao Jingdong Chen Haifeng Li, [1ex] Central South University Huazhong University of Science and Technology · 2026

Read The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence? by Zhaoyang Zhang Run Shao Dongyue Wu Jiajie Teng, Chao Tao Jingdong Chen Haifeng Li, [1ex] Central South University Huazhong University of Science and Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09352.txt.

Introduction

Zhang and six colleagues ask why networks trained apart in different senses end up sharing representations. Past evidence uses similarity scores that work the same both ways, spotting sharing but missing its direction.

The team introduces one-way sharing analysis with cycle-kNN, an uneven alignment score, used on dozens of unimodal models trained apart, covering point clouds, vision, and language. They find a steady one-way tilt: senses other than language drift toward language's local layout far more than the reverse, across all model families and scales, yet same-both-ways measures miss it entirely. A look at the mechanism traces the tilt to uneven feature density, with language representations in the tightest regions of the space. Under the Information Bottleneck view, squeezing representations drives them toward discrete, compositional structures typical of language. That is the Wittgensteinian Representation Hypothesis: language's meaning structure is the end point of multimodal sharing.

Catalog record

CREATORZhaoyang Zhang Run Shao Dongyue Wu Jiajie Teng, Chao Tao Jingdong Chen Haifeng Li, [1ex] Central South University Huazhong University of Science and Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09352

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence? Zhaoyang Zhang Run Shao Dongyue Wu Jiajie Teng, Chao Tao Jingdong Chen Haifeng Li, [1ex] Central South University Huazhong University of Science and Technology Abstract Understanding why independently trained neural networks from different modalities converge toward shared representations, and where this convergence leads, remains an open question in representation learning. All existing evidence relies on symmetric similarity measures, which can detect convergence but are structurally blind to its direction. We introduce directional convergence analysis using cycle-kNN , an asymmetric alignment measure, applied across dozens of independently trained unimodal models spanning point clouds, vision, and language. We uncover a consistent directional asymmetry: non-language modalities move toward the neighborhood structure of language significantly more than the reverse, and this pattern holds across all model families and scales—yet is entirely invisible to symmetric measures. Mechanistic analysis traces the directionality to feature density asymmetry, whereby language representations occupy the most compact regions of representational space. The Information Bottleneck framework provides a principled interpretation: optimization under compression drives representations toward discrete, compositional structures characteristic of language. We formalize this as the Wittgensteinian Representation Hypothesis : the semantic structure of language is the asymptotic attractor of multimodal representation convergence. 1 Introduction Understanding why representations from different modalities tend to converge, and identifying the endpoint of such convergence, is a fundamental question in multimodal representation learning. The Platonic Representation Hypothesis (PRH) ( 14 ) posits that independently trained models converge toward shared representational structures, an observation reinforced by theoretical work ( 23 ; 50 ; 40 ) and empirical demonstrations of cross-modal alignment without paired data ( 35 ; 25 ) . However, all empirical evidence relies on symmetric similarity measures —CKA ( 19 ) , mutual kNN ( 14 ) , RSA ( 21 ) —which by construction satisfy s(\mathbf{X},\mathbf{Y})=s(\mathbf{Y},\mathbf{X}) . No prior work employs an asymmetric measure to investigate convergence direction ( 18 ) . Like a ball rolling into a valley, if representation convergence possesses directional structure, symmetric measures are fundamentally blind to it. Figure 1 : Overview of directional convergence analysis. (a) Symmetric measures (e.g., CKA) detect convergence but cannot reveal its direction. (b) cycle-kNN is inherently asymmetric; analyzing both directions reveals a consistent directional bias: vision representations approach language more than the reverse ( \Delta=+0.010 , p<0.05 , across all 22 model pairs). (c) Three modalities span an abstraction hierarchy; representational compactness ( D ) increases toward language. WRH posits that language is the attractor of multimodal convergence. We fill this gap with directional convergence analysis (Figure 1 ). Our key insight is that cycle-kNN —a known alignment measure whose inherent asymmetry has been consistently overlooked ( 14 ) —naturally encodes convergence direction. We systematically evaluate 58 independently trained unimodal models spanning three modalities along an abstraction hierarchy : 3D point clouds \rightarrow 2D vision \rightarrow language. Our core findings: • Convergence has direction. cycle-kNN reveals consistent asymmetry: vision and point cloud representations approach language significantly more than the reverse—invisible to symmetric measures on the same pairs. • Language shows maximal intra-modality consensus , consistent with proximity to the attractor. • Directionality is scale-invariant. \Delta>0 toward language holds for every vision model (22/22), every point cloud model (7/7), and nearly every language model across ten families spanning four orders of magnitude. • Feature density asymmetry as geometric mechanism. Language representations are among the most compact; the Information Bottleneck framework provides a principled interpretation. These findings carry broad implications for the field. If convergence is directional, then the attractor is not a generic shared structure but a specific one—language—which suggests that the discrete, compositional nature of language acts as a natural compression target. This perspective sharpens a question left open by PRH: if representations converge, where do they converge to? Our analysis reveals that convergence is not mutual in any meaningful sense—it is asymmetric, and the asymmetry follows the abstraction hierarchy. Moreover, the phenomenon is remarkably robust: it holds across ten model families spanning four orders of magnitude in parameters and is confirmed by k -sensitivity analysis across multiple neighborhood sizes. Methodologically, our work highlights a systematic blind spot: the exclusive reliance on symmetric measures in representation comparison has left convergence direction completely unexamined. We do not propose a new metric; rather, we recognize that cycle-kNN ’s asymmetry—catalogued by 14 but never analyzed—encodes directional structure, and we build an analysis framework to extract it. This “directional convergence analysis” framework is useful independently of any specific hypothesis about the attractor. Based on these findings, we propose the Wittgensteinian Representation Hypothesis (WRH): the semantic structure of language constitutes the asymptotic attractor of multimodal representation convergence. The name invokes Wittgenstein’s dictum “The limits of my language mean the limits of my world” ( Tractatus 5.6), reinterpreted here as a statement about representation learning: the structural constraints imposed by language define the boundary toward which all perceptual representations converge under sufficient optimization. Unlike the Platonic Representation Hypothesis, which is agnostic about the convergence endpoint, WRH makes a falsifiable directional prediction—and our experiments confirm it across all 58 models. The remainder of this paper is organized as follows. Section 2 defines cycle-kNN and its directional interpretation. Section 3 presents empirical findings on directional asymmetry, intra-modality consensus, and scale invariance. Section 4 provides mechanistic analysis. Section 5 formally states WRH and positions it within the hypothesis landscape, and Sections 6 – 7 discuss related work and conclude. 2 Preliminaries 2.1 Cycle-kNN: An Asymmetric Alignment Measure Let \mathbf{X}\in\mathbb{R}^{N\times d_{1}} and \mathbf{Y}\in\mathbb{R}^{N\times d_{2}} denote representations of N shared stimuli. Standard alignment measures—CKA ( 19 ) , mutual kNN ( 14 ) , RSA ( 21 ) —are symmetric by construction: s(\mathbf{X},\mathbf{Y})=s(\mathbf{Y},\mathbf{X}) . They measure alignment degree but cannot reveal its direction . cycle-kNN ( 14 ) computes alignment through a two-step nearest-neighbor cycle: \displaystyle\text{{cycle-kNN} }(\mathbf{X}\to\mathbf{Y};k)=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\left[i\in\mathrm{kNN}_{\mathbf{X}}\!\big(\mathrm{kNN}_{\mathbf{Y}}(i)\big)\right] (1) where \mathrm{kNN}_{\mathbf{Y}}(i) finds the k nearest neighbors of i in space \mathbf{Y} , and \mathrm{kNN}_{\mathbf{X}}(\cdot) returns to space \mathbf{X} . Our key observation : although 14 note that cycle-kNN is the only asymmetric metric among eight they catalog, its asymmetry has never been analyzed. We show it carries structured directional information. The asymmetry arises because when the first-hop space \mathbf{Y} has more coherent neighborhoods, semantically similar samples cluster tightly, making the return hop more likely to succeed: \displaystyle\text{{cycle-kNN} }(\mathbf{X}\!\to\!\mathbf{Y})>\text{{cycle-kNN} }(\mathbf{Y}\!\to\!\mathbf{X})\;\Longrightarrow\;\mathbf{Y}\text{'s neighborhoods are more coherent.} (2) We define the directional gap \Delta(\mathbf{X},\mathbf{Y})=\text{{cycle-kNN} }(\mathbf{X}\to\mathbf{Y})-\text{{cycle-kNN} }(\mathbf{Y}\to\mathbf{X}) . A consistently positive \Delta indicates that \mathbf{Y} ’s structure is a more stable reference frame. Robustness is confirmed through synthetic experiments with matched dimensionality (Appendix C ), k -sensitivity analysis ( k=1\ldots 50 ), and permutation tests. 2.2 Experimental Setup Models. We assemble 58 independently trained models spanning three modalities along an abstraction hierarchy : 7 point cloud models (PointMAE ( 30 ) , PointBERT ( 49 ) , PointGPT ( 7 ) , ReCon++ ( 31 ) ; 16M–300M params), 22 vision models across supervised, self-supervised, and contrastive paradigms (ViT ( 9 ) , MAE ( 13 ) , DINOv2 ( 29 ) , DINOv3 ( 5 ) , CLIP ( 34 ) ; 5.7M–7B), and 29 language models from five families (BLOOMZ ( 26 ) , LLaMA ( 44 ) , Qwen2.5 ( 32 ) , Qwen3 (

Read the complete text