How Language Directions Align with Token Geometry in Multilingual LLMs Conference: The Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates CCS: Computing methodologies Information extraction CCS: Computing methodologies Language resources
JaeSeong Kim, Suan Lee · 2025
Read How Language Directions Align with Token Geometry in Multilingual LLMs Conference: The Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates CCS: Computing methodologies Information extraction CCS: Computing methodologies Language resources by JaeSeong Kim, Suan Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16693.txt.
Introduction
JaeSeong Kim and Suan Lee map how language identity is organized inside multilingual language models, probing six models across all 268 transformer layers with linear and nonlinear probes. Their new token-language alignment analysis tracks how language directions in the representation space relate to vocabulary embeddings, layer by layer.
Language information separates sharply in the very first transformer block, jumping by about seventy-six percentage points, and stays almost fully linearly separable through the whole depth. Alignment between language directions and word embeddings tracks the training data's language mix. Models trained with Chinese align Chinese far more strongly than English-centric models do, a more than fourfold structural imprinting effect. The authors argue models distinguish languages through latent structure shaped by the corpus, not surface script, with lessons for data composition and fairness.
Catalog record
| CREATOR | JaeSeong Kim, Suan Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-16693
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Language Directions Align with Token Geometry in Multilingual LLMs Conference: The Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates CCS: Computing methodologies Information extraction CCS: Computing methodologies Language resources JaeSeong Kim, Suan Lee Abstract . Multilingual LLMs demonstrate strong performance across diverse languages, yet there has been limited systematic analysis of how language information is structured within their internal representation space and how it emerges across layers. We conduct a comprehensive probing study on six multilingual LLMs, covering all 268 transformer layers, using linear and nonlinear probes together with a new Token–Language Alignment analysis to quantify the layer-wise dynamics and geometric structure of language encoding. Our results show that language information becomes sharply separated in the first transformer block (+76.4±8.2%p from Layer 0→1) and remains almost fully linearly separable throughout model depth. We further find that the alignment between language directions and vocabulary embeddings is strongly tied to the language composition of the training data . Notably, Chinese-inclusive models achieve a ZH Match@Peak of 16.43%, whereas English-centric models achieve only 3.90%, revealing a 4.21× structural imprinting effect. These findings indicate that multilingual LLMs distinguish languages not by surface script features but by latent representational structures shaped by the training corpus . Our analysis provides practical insights for data composition strategies and fairness in multilingual representation learning. All code and analysis scripts are publicly available at: https://github.com/thisiskorea/How-Language-Directions-Align-with-Token-Geometry-in-Multilingual-LLMs . 1. Introduction Multilingual large language models (LLMs) have become essential infrastructure for global information access, achieving human-level performance across more than 50 languages. Models such as Llama-3.1 ( 7 ) and Qwen2.5 ( 11 ) demonstrate strong capabilities not only in English but also in Spanish, Chinese, Arabic, and other languages, suggesting the emergence of a universal semantic space shared across languages ( 10 ; 5 ) . However, a persistent gap remains: most models perform best in English, while accuracy in non-English languages is often 10–30% lower ( 9 ) . This gap is commonly attributed to the English-centric imbalance in pretraining data . English constitutes 70–80% of typical web-scale corpora ( 3 ) , and limited exposure to other languages leads to performance degradation. Yet a fundamental question remains unanswered: does the data distribution merely modulate performance, or does it fundamentally shape the geometry of the internal representation space? If it is merely performance-dependent, post-hoc interventions such as fine-tuning or vocabulary adjustments may suffice; if it is structurally embedded, redesigning the pretraining stage is necessary. Positioning within prior work. Prior studies have shown that multilingual models learn shared cross-lingual spaces ( 10 ; 4 ) . Early models such as mBERT ( 6 ) and XLM-R ( 5 ) demonstrated zero-shot transfer across languages, and more recent models such as Llama-3 and Qwen2.5 further strengthened cross-lingual capabilities with larger and more diverse corpora. Meanwhile, probing-based interpretability studies ( 1 ; 8 ; 2 ) examined how linguistic and syntactic information is encoded across layers, but did not systematically address (i) whether linear separability of language information is universal, (ii) at which depth language directions emerge , and (iii) how pretraining language distributions are imprinted into the model’s internal geometry. Our work fills this gap. This study investigates two complementary research questions: (1) Where and how is language information encoded within transformer layers? (2) Does the pretraining language distribution structurally reshape the geometry of multilingual representations? To answer these questions, we conduct a comprehensive probing study across all 268 transformer layers of six representative multilingual LLMs (Llama-3.1-8B, Qwen2.5-7B, Qwen2.5-Math-7B, OpenMath2-8B, OpenR1-7B, GPT-OSS-20B). We train linear and nonlinear probes and introduce a new analysis, Token–Language Alignment , across five typologically diverse languages (EN, ES, ZH, FR, DE), yielding a total of 2,680 independent experiments . Key Findings. (1) Universal Linear Separability. Across all models, language information is encoded in linearly accessible subspaces: linear probes achieve 99.8±0.1% accuracy on average, with only a 0.58±0.12%p gap relative to MLP probes (p<0.001). Remarkably, this linear structure emerges immediately at the first transformer block (Layer 0→1), with a +76.4±8.2%p jump, suggesting that language separation is an architecture-inherent mechanism. (2) Structural Imprinting. Token–Language Alignment reveals that the alignment between language directions and vocabulary is strongly tied to the pretraining language distribution. English-centric models (EN>80%) show 69.05% alignment for English but only 3.90% for Chinese, whereas Chinese-inclusive models (ZH≈20%) show a markedly higher 16.43% alignment for Chinese (4.21× increase). This indicates that the geometry of representation space is structurally shaped by the pretraining distribution , beyond its effects on task performance. (3) Typological Effects and Complex Alignment Patterns. Chinese reaches its maximal separability in deeper layers (Layer 5.2±0.8), whereas Spanish and German converge earlier (Layer 2.5±0.4). The overall Match@Peak remains low (15.0±0.3%), showing that language directions capture abstract, multi-factorial features beyond simple token–language matching, including script, lexical frequency, and typological structure. Implications. These findings suggest that achieving fairness and balance in multilingual LLMs requires careful design of pretraining data composition . Structural imprinting implies that post-hoc adjustments cannot fundamentally alter the underlying geometry. Moreover, Match@Peak provides a quantitative diagnostic tool for evaluating the effects of data rebalancing on representation richness. 2. Methodology We analyze how language information is encoded across all 268 transformer layers of six multilingual LLMs, and how such structure aligns with the pretraining language distribution. Our methodology consists of four parts: (1) experimental setup, (2) linear and nonlinear probing, (3) Token–Language Alignment, and (4) evaluation and statistical analysis. 2.1. Experimental Setup We study six multilingual LLMs whose pretraining corpora differ in their language composition: • English-centric models: Llama-3.1-8B, OpenMath2-8B • Chinese-inclusive models: Qwen2.5-7B, Qwen2.5-Math-7B, OpenR1-7B • Balanced baseline: GPT-OSS-20B • Languages: Following prior probing studies, we evaluate five XNLI languages (EN, ES, FR, DE, ZH), using 5k training and 2.5k validation sentences per language. 2.2. Probing Methodology For each model and each layer \ell , we use the hidden vector of the final token, \mathbf{h}^{(\ell)}\in\mathbb{R}^{d}, as the sentence representation. Linear probe. A linear classifier is applied to LayerNorm-normalized representations: f_{\mathrm{lin}}(\mathbf{h})=W_{c}\cdot\mathrm{LN}(\mathbf{h})+b_{c}, trained with cross-entropy loss to predict one of the five languages. MLP probe. To compare linear and nonlinear capacity, we additionally train an MLP: f_{\mathrm{mlp}}(\mathbf{h})=W_{2}\cdot\mathrm{ReLU}\!\left(W_{1}\cdot\mathrm{LN}(\mathbf{h})\right). LayerNorm is applied in both probes to remove inter-layer scale differences and measure pure linear separability of language information. Both probes are trained independently for each (model, layer, seed, probe type) using AdamW ( \mathrm{lr}=10^{-3} ), batch size 128, and 3 epochs with early stopping. 2.3. Token–Language Alignment To quantify how learned language directions relate to the LM head vocabulary, we compute cosine similarity between each probe-learned language direction \mathbf{w}^{(\ell)}_{L} and each vocabulary embedding \mathbf{e}_{v} : \mathrm{sim}(v,L,\ell)=\cos\!\left(\mathbf{e}_{v},\,\mathbf{w}^{(\ell)}_{L}\right). Each token v is assigned to the language direction with highest similarity. For each language L , we compute three alignment metrics: • PeakDepth L : The normalized layer index where \mathrm{VocabShare}(\ell,L) is maximized, indicating the depth at which language L is most strongly expressed. • PeakVocab L : The maximum value of \mathrm{VocabShare}(\ell,L) across layers, representing how dominant language L is within the vocabulary space. • Match@Peak L : Among tokens assigned to language L at its peak layer, the percentage whose decoded text belongs to language L , determined using Unicode and diacritic rules (e.g., CJK blocks for ZH, accent-sensitive characters for ES/FR/DE). Higher values indicate stronger alignment between