Language-Specific Latent Process Hinders Cross-Lingual Performance

Zheng Wei Lim Alham Fikri Aji Trevor Cohn, [3mm] The University of Melbourne MBZUAI Google · 2025

Read Language-Specific Latent Process Hinders Cross-Lingual Performance by Zheng Wei Lim Alham Fikri Aji Trevor Cohn, [3mm] The University of Melbourne MBZUAI Google (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13141.txt.

Introduction

Zheng Wei Lim and co-authors probe why large language models answer unevenly across languages, in this 2025 paper. The same question in two tongues can draw different answers, even though models can transfer knowledge across languages. The team measured how alike the models' inner representations are between languages. They also used a lens tool to peek at the hidden steps behind each answer.

Models stumbled when their inner states for a language drifted from the shared meaning space. Bigger models strayed further from that shared space, yet still fetched cross-language knowledge better. Steering a small model's hidden process back toward the shared space improved its reasoning in other languages. Output also grew more consistent with English. The findings point to a concrete fix for multilingual gaps.

Catalog record

CREATORZheng Wei Lim Alham Fikri Aji Trevor Cohn, [3mm] The University of Melbourne MBZUAI Google
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13141

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Language-Specific Latent Process Hinders Cross-Lingual Performance Zheng Wei Lim Alham Fikri Aji Trevor Cohn, [3mm] The University of Melbourne MBZUAI Google Abstract Large language models (LLMs) are demonstrably capable of cross-lingual transfer, but can produce inconsistent output when prompted with the same queries written in different languages. To understand how language models are able to generalize knowledge from one language to the others, we apply the logit lens to interpret the implicit steps taken by LLMs to solve multilingual multi-choice reasoning questions. We find LLMs predict inconsistently and are less accurate because they rely on subspaces of individual languages, rather than working in a shared semantic space. While larger models are more multilingual, we show their hidden states are more likely to dissociate from the shared representation compared to smaller models, but are nevertheless more capable of retrieving knowledge embedded across different languages. Finally, we demonstrate that knowledge sharing can be modulated by steering the models’ latent processing towards the shared semantic space. We find reinforcing utilization of the shared space improves the models’ multilingual reasoning performance, as a result of more knowledge transfer from, and better output consistency with English. 1 Introduction Humans have an innate ability to apply common knowledge and perform reasoning skills consistently across different languages, as the cognitive functions involved are largely independent of the languages they are learned in [ 5 , 4 ] . Modern large language models (LLMs) are trained predominantly in English [ 24 , 8 ] and Chinese [ 35 ] . To some degree, LLMs demonstrate cross-lingual capabilities [ 27 ] , but the performance gap between languages remains [ 15 , 13 ] , and the models struggle to maintain consistent responses when they are probed by the same queries in different languages [ 20 , 12 , 7 ] . One possible explanation is that language models tend to develop language-specific neurons, rather than sharing them among multiple languages [ 29 ] . Nevertheless, language-specific neurons have never been shown to cause inconsistency in LLMs directly. Other work suggests that LLMs operate in a shared subspace closest to the English tokens, before projecting back to other languages in higher layers, for predictions in multilingual and multi-modal tasks [ 32 , 34 , 41 ] . These findings, however, fail to explain the lack of consistency across languages for the largest models [ 7 ] — otherwise, model outputs would converge toward English-prompted responses as multilingual performance improves. This work investigates how language models generalize knowledge to multiple languages. We experiment on Gemma 2 and Qwen 2.5 models and show that: Utilization of shared semantic space in multilingual processing facilitates knowledge transfer. We first evaluate LLMs’ cross-lingual consistency and transfer on multi-choice queries. We find larger LLMs are indeed more accurate, but present large performance gaps, and remain inconsistent with poor cross-lingual transfer. We then investigate the intermediate steps taken implicitly by LLMs in solving the queries, and discover that models produce more divergent responses when operating in subspaces of different languages, rather than a shared internal representation that is closest to English hidden representation when processing multilingual prompts. We present evidence demonstrating a strong link between this phenomenon and the transfer performance of LLMs. Implicit processing in English is more stable and accurate. We interpret the hidden representations of LLMs using the logit lens [ 18 , 32 , 42 , 34 ] , which project intermediate hidden layers to the vocabulary space, to compute the latent distribution of answer choices. LLMs are capable of distinguishing multilingual answer choices in their hidden states, as indicated by our observation that the internal representations are closer to the embedding of the correct answers, relative to the other choices. Our results show the implicit process in English is more stable and more accurate than the process in the native languages of the prompts. Nonetheless, larger models tend to reason in the native language of the prompt. In line with previous work, we find models of smaller scale work predominantly in the English shared space. Larger models, however, dissociate their internal states from the dominant semantic space, and move towards the individual subspaces of the native languages. While we have shown this phenomenon to undermine multilingual reasoning performance, the impact is less pronounced on larger models. We attribute this to their capability of encoding and retrieving information in different languages independently with significantly more parameters. Inducing processing in the shared semantic space improves cross-lingual transfer and model performance in reasoning tasks. Building on current understanding of knowledge transfer, how can we improve cross-lingual consistency while retaining performance of individual languages? The final part of this work demonstrates that knowledge sharing and consistency can be modulated by reinforcing or diminishing processing in the English latent space. To this end, we introduce cross-lingual activation steering, inspired by prior work [ 30 , 23 ] . We find adding a steering vector towards English produces more English-like predictions, leading to performance gains in smaller models; whereas adding an opposite vector results in more divergent output, with a significant dip in performance. In summary, our work provides an account of the internal multilingual processing of LLMs and explains the limitations of current models in knowledge transfer. To further validate our hypotheses we explore cross-lingual activation steering. We recommend future work to develop more comprehensive methods that encode knowledge of different languages in the shared semantic space to maximize transfer between languages. 2 Methodology 2.1 Cross-lingual consistency and knowledge transfer We define an LLM to be cross-lingually consistent when it produces the same output to the same queries written in different languages. Unlike Ifergan et al. 2024 , our measure of consistency is based upon multiple-choice question-answering tasks. To reduce language variability in the output space, we formulate the consistency between a language pair by their similarity in answer ranks, given by an LLM when prompted with the same questions and answer choices in their respective languages. Formally, for each of i\in I questions in language l , q_{i,l}\in Q_{l} , paired with j\in\{1..J\} answer choices [a_{i,l}^{1},...,a_{i,l}^{J}] , we rank the answer choices by their output probability p(a_{i,l}^{j}|q_{i,l}) given the question. We then combine the ranks k(a_{i,l}^{j}) of each answer within the set of choices: \displaystyle\mathbf{k}_{l} \displaystyle=\langle k(a_{1,l}^{1}),...,k(a_{1,l}^{J}),k(a_{2,l}^{1}),...,k(a_{I,l}^{J})\rangle. Thus, the pairwise consistency between languages l_{1} and l_{2} is defined as the Spearman’s rank correlation coefficient between the rank vectors: \mathrm{consistency}({l_{1},l_{2}})=\rho(\mathbf{k}_{l_{1}},\mathbf{k}_{l_{2}}). (1) Not all transfer between languages is beneficial—a model can be cross-lingually consistent but make many incorrect predictions. To approximate the rate of correct answers in a language propagates to another, we adapt our approach from Ifergan et al. 2024 and estimate the positive transfer from l_{1} to l_{2} by the ratio of shared accurate responses to all accurate responses in l_{1} : \displaystyle\mathrm{tr}^{+}(l_{1},l_{2})=|F_{l_{1}}\cap F_{l_{2}}|/|F_{l_{1}}|. (2) F_{l} denotes the set of questions in l that the model answers correctly. Conversely, \neg F_{l} refers to the set of questions in l where the model responses incorrectly, and \neg F_{l\>\cap\>m} the questions in l and m where the model return the same incorrect responses. We compute the negative transfer from l_{1} as: \mathrm{tr}^{-}(l_{1},l_{2})=|\neg F_{l_{1}\>\cap\>l_{2}}|/|\neg F_{l_{1}}|. (3) We note that transfer rates defined in Equations 2 and 3 are

Read the complete text