Understanding LLMs’ Cross-Lingual Context Retrieval: How Good It Is And Where It Comes From
Changjiang Gao ♣ \clubsuit, Hankun Lin ♣ \clubsuit, Xin Huang ◇ \Diamond · 2025
Read Understanding LLMs’ Cross-Lingual Context Retrieval: How Good It Is And Where It Comes From by Changjiang Gao ♣ \clubsuit, Hankun Lin ♣ \clubsuit, Xin Huang ◇ \Diamond (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10906.txt.
Introduction
Language models can fetch facts across languages, and Changjiang Gao and co-authors measure how in a 2025 paper. More than forty models faced reading tests in twelve languages, answering in one tongue from passages in another. Post-trained open models matched top closed ones at the task.
Two phases split the skill: grasping the question forms in pre-training, while fetching the answer is shaped later. Steadier phasing tracks better scores. The bottleneck sits in the last layers during the fetching phase. Knowing where cross-lingual skill lives helps builders fix it without retraining whole models.
Catalog record
| CREATOR | Changjiang Gao ♣ \clubsuit, Hankun Lin ♣ \clubsuit, Xin Huang ◇ \Diamond |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10906
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding LLMs’ Cross-Lingual Context Retrieval: How Good It Is And Where It Comes From Thanks: Corresponding author Changjiang Gao ♣ \clubsuit, Hankun Lin ♣ \clubsuit, Xin Huang ◇ \Diamond Abstract Cross-lingual context retrieval (extracting contextual information in one language based on requests in another) is a fundamental aspect of cross-lingual alignment, but the performance and mechanism of it for large language models (LLMs) remains unclear. In this paper, we evaluate the cross-lingual context retrieval of over 40 LLMs across 12 languages, using cross-lingual machine reading comprehension (xMRC) as a representative scenario. Our results show that post-trained open LLMs show strong cross-lingual context retrieval ability, comparable to closed-source LLMs such as GPT-4o, and their estimated oracle performances greatly improve after post-training. Our mechanism analysis shows that the cross-lingual context retrieval process can be divided into two main phases: question encoding and answer retrieval, which are formed in pre-training and post-training respectively. The phasing stability correlates with xMRC performance, and the xMRC bottleneck lies at the last model layers in the second phase, where the effect of post-training can be evidently observed. Our results also indicate that larger-scale pretraining cannot improve the xMRC performance. Instead, larger LLMs need further multilingual post-training to fully unlock their cross-lingual context retrieval potential. 1 1 1 Our code and results are available at https://github.com/NJUNLP/Cross-Lingual-Context-Retrieval (a) en-x (b) x-x Figure 1: Examples of our en-x and x-x testing scenarios. The figures show examples in English (en), German (de), and Chinese (zh). 1 Introduction Since the rise of Large language models (LLMs), many models have demonstrated their strong capability in various NLP tasks Chang et al. (2024) , e.g. ChatGPT 2 2 2 https://chatgpt.com , Claude 3 3 3 https://claude.ai , Gemini Gemini Team et al. (2024) , LLaMA Grattafiori et al. (2024) , Qwen Qwen et al. (2025) , DeepSeek DeepSeek-AI et al. (2024b) , etc. However, due to the dominance of English training data, most of these LLMs show their best performance in English Lai et al. (2023b) . To improve their performance and efficiency in non-English languages, cross-lingual alignment has become a major research topic for multilingual LLMs Qi et al. (2023) ; Wang et al. (2024) , which encourages LLMs to share capabilities across languages. For example, given requests with the same semantics but in different languages, LLMs should give consistent answers. Such alignment has been shown challenging when the task requires recalling trained knowledge Gao et al. (2024) ; Hu et al. (2025) . Thus, a follow-up question is that, when the knowledge is presented in the context in one language (e.g. English), could LLMs retrieve it when answering requests in the another language? Note that this is different from multilingual context retrieval (the context and the request are in the same language) and cross-lingual information retrieval (retrieving queried text from a database). However, evaluation and mechanism analysis of this ability in LLMs are not fully explored. In this paper, we evaluate cross-lingual context retrieval of SOTA multilingual LLMs, and analyze the mechanism of it. We use cross-lingual machine reading comprehension (xMRC) Cui et al. (2019) as a simplified but representative scenario because: (1) It is a representative NLP task where models need to retrieve cross-lingual knowledge within the context; (2) The target knowledge to be retrieved is entirely within the context, so minimal extra knowledge recall is needed; (3) The model only has to copy part of the context as the answer, simplifying evaluation and mitigating other factors such as low-resource language generation errors. Furthermore, we conduct in-depth analysis of the performance bottleneck, oracle performance, and mechanism of LLMs doing the xMRC task, using suitable tools and specially designed metrics. Specifically, we propose a hypothesis of two-phased xMRC process in LLMs, and verify our hypothesis with experimental evidence. Our main findings are: • Post-trained open-source LLMs, especially 7-9B versions, show strong xMRC ability, catching up with closed-source models. Larger models show higher English MRC performance, but larger gap between English and non-English. • Post-training significantly improves the estimated oracle performance to almost saturate in all tested languages, setting space for improvement of real performance, while the effect of extended pretraining is minor. • The xMRC process can be divided into two phases within the model: question encoding and answer retrieval. The former forms in pre-training, and its stability correlates with the base model capability, as well as xMRC performance; the latter forms in post-training, and serves as a bottleneck. • One reason for the larger language gap observed on larger post-trained open-source LLMs might be insufficient and language-biased post training. 2 Methods 2.1 Evaluation methods 2.1.1 Scenarios We use English to non-English (en-x) as a typical and common cross-lingual scenario, where the context and answer are in English, and the question is in non-English, because it avoids the requirement of non-English generation fluency that is not related to cross-lingual context retrieval, and keeps the answers always in English which makes answer verification easier and fairer. Meanwhile, non-English monolingual (x-x) is used as comparison to ablate the effect of non-English understanding. Figure 1 shows examples of the testing scenarios. 2.1.2 Metrics Performance metrics. We use the F1 score to evaluate xMRC performance, as adopted by XQuAD Artetxe et al. (2020) . Exact match (EM) is not used because it tends to under-estimate the performance due to unnecessary source text in the answers (see Appendix A.1 ). Cross-lingual performance metrics. To measure performance alignment between English and non-English, we calculate the average performance on all the non-English languages divided by English (en-x/en-en, x-x/en-en). 2.1.3 Performance bottleneck analysis Error type ablation. Based on observation of model responses, we distinguish error types as: • Language : answering in x instead of en ; • Generation : generation errors including gibberish, refusal and blank answers; • Content : meaningful but incorrect answers in the correct language; For example, the language error rate for test setting en-x is calculated as: E_{\mathrm{lang}}(\mathrm{en,x})=\sum_{\mathclap{(r,a)\in W_{\mathrm{en,x}}}}\frac{\scalebox{0.9}{$\mathbb{I}(\mathrm{Lang}(r)=\mathrm{en})\mathbb{I}(\mathrm{Lang}(a)=\mathrm{x})$}}{|W_{\mathrm{en,x}}|} where r is the reference answer, a is the model prediction, and \mathrm{Lang}(\cdot) is a language detector for given text. Meanwhile, a generation error rate, e.g. gibberish, will be: E_{\mathrm{gib}}(\mathrm{en,x})=\sum_{\mathclap{(r,a)\in W_{\mathrm{en,x}}}}\frac{\scalebox{0.9}{$\mathbb{I}(\mathrm{Type}(r)=\mathrm{Gibberish})$}}{|W_{\mathrm{en,x}}|} where \mathrm{Type}(\cdot) is an LLM-based error type classifier for generation errors. Oracle performance estimation The xMRC score can be affected by errors in the generation process, under-estimating LLMs’ cross-lingual context retrieval ability. To ablate this effect, we estimate the oracle retrieval performance of LLMs by perturbation-based attribution 4 4 4 We use Captum to perform the attribution ( https://github.com/pytorch/captum ) on contextual sentences or spans. If the sentence/span with the correct answer receives the largest attribution score, we consider the model’s oracle retrieval on this testing sample correct. 2.2 Mechanism analysis methods 2.2.1 Layer-wise attribution to reflect forward process To better understand the forward process of LLMs performing xMRC, we need layer-wise attribution to observe information flow from context parts into the residual stream Voita et al. (2024) at each layer. Here we adopt AttentionLRP Achtibat et al. (2024) , an attribution method based on Taylor Decomposition on attention, FFN and normalization modules of LLMs, that can calculate the relevance of token representations in each layer to the output. Based on the attribution results, we define the Major Relevance Depth (MRD) to estimate the maximum