Language Models’ Factuality Depends on the Language of Inquiry

Tushar Aggarwal, Kumar Tanmay, Ayush Agrawal, Kumar Ayush, Hamid Palangi, Paul Pu Liang · 2025

Read Language Models’ Factuality Depends on the Language of Inquiry by Tushar Aggarwal, Kumar Tanmay, Ayush Agrawal, Kumar Ayush, Hamid Palangi, Paul Pu Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17955.txt.

Introduction

Tushar Aggarwal and co-authors show that a language model can know a fact in one language and lose it in another. A model may name a person's home country correctly in Arabic yet fail the same question in English or Swahili. To measure the problem, the team built a benchmark of 10,000 country facts in 13 languages and devised three scores for recall and for how well knowledge crosses languages.

Even the best current models stumbled, above all at carrying knowledge across languages, so their answers changed with the language of the question. The authors argue models should learn which of their language selves is most reliable on a topic and lean on it. They release the benchmark for others to use.

Catalog record

CREATORTushar Aggarwal, Kumar Tanmay, Ayush Agrawal, Kumar Ayush, Hamid Palangi, Paul Pu Liang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Language Models’ Factuality Depends on the Language of Inquiry Tushar Aggarwal, Kumar Tanmay, Ayush Agrawal, Kumar Ayush, Hamid Palangi, Paul Pu Liang Abstract Multilingual language models (LMs) are expected to recall factual knowledge consistently across languages, yet they often fail to transfer knowledge between languages even when they possess the correct information in one of the languages. For example, we find that an LM may correctly identify Rashed Al Shashai as being from Saudi Arabia when asked in Arabic, but consistently fails to do so when asked in English or Swahili. To systematically investigate this limitation, we introduce a benchmark of 10,000 country-related facts across 13 languages and propose three novel metrics—Factual Recall Score, Knowledge Transferability Score, and Cross-Lingual Factual Knowledge Transferability Score—to quantify factual recall and knowledge transferability in LMs across different languages. Our results reveal fundamental weaknesses in today’s state-of-the-art LMs, particularly in cross-lingual generalization where models fail to transfer knowledge effectively across different languages, leading to inconsistent performance sensitive to the language used. Our findings emphasize the need for LMs to recognize language-specific factual reliability and leverage the most trustworthy information across languages. We release our benchmark and evaluation framework to drive future research in multilingual knowledge transfer. The data and codes are available at this link . † † footnotemark: † † footnotetext: *equal contribution. † † footnotetext: Corresponding authors: [email protected], [email protected], [email protected] 1 Introduction Figure 1: Illustratation of the cross-lingual factual knowledge transferability issue across linguistic knowledge clouds in LMs. The model correctly recalls that Rashed Al Shashai is from Saudi Arabia when queried in Arabic, but fails to retrieve this fact in English and Swahili, highlighting that factual knowledge is often stored in language-specific silos. Large Language Models (LLMs) are often perceived as vast knowledge reservoirs, capable of recalling factual information across multiple languages Wang et al. (2024) . However, what if their knowledge is locked within linguistic boundaries and unable to be transferred across languages? Despite advancements in multilingual LMs such as Llama Touvron et al. (2023a) ; Dubey et al. (2024) , Gemma Team et al. (2024a) , DeepSeek DeepSeek-AI et al. (2024) , and Phi Abdin et al. (2024) ; Li et al. (2023) , our study reveals a striking asymmetry in their factual recall across languages: consider the example in Figure 1 , where an LM is tasked with a simple factual query: “ Rashed Al Shashai is from which country? ” When asked in Arabic, several state-of-the-art LMs correctly generate the response: “ Saudi Arabia .” However, when posed in English, Hindi, or Swahili, the same models fail to recall the fact. This example suggests that models can correctly retrieve country-specific facts in the language associated with that country but struggle to do so in others. This raises a critical question—do these models truly internalize and transfer factual knowledge across languages, or do they merely encode isolated linguistic silos? This limitation has significant implications for multilingual AI development and real-world applications. Many LM-based systems—such as retrieval-augmented generation (RAG) pipelines, multilingual search engines, and cross-lingual reasoning models—assume that factual knowledge is consistently available and transferable across languages. Our findings reveal that LMs often rely on language-specific memorization rather than true cross-lingual knowledge generalization. This over-reliance can introduce biases, inconsistencies, and reliability issues in multilingual AI applications Chua et al. (2024) . To systematically analyze the factual inconsistencies, we introduce a carefully curated dataset comprising country-related facts translated into 13 languages. This benchmark evaluates LMs on multiple dimensions— factual recall, in-context recall, and counter-factual context adherence —across high-, medium-, and low-resource languages. This benchmark comprises of 802 instances for factual recall, 156 instances for In-context recall, and 1404 instances for counter-factual context adherence as shown in Table 1 . Factual recall assesses the LM’s ability to recall country-specific facts consistently across multiple languages. We evaluate factual recall using three metrics: (a) Factual Recall Score (FRS) : Measures how accurately a model recalls a fact in a given language, (b) Knowledge Transferability Score (KTS) : Quantifies how well factual knowledge is transferred across languages, and (c) Cross-Lingual Factual Knowledge Transferability (X-FaKT) Score : Combines the assessment of factual recall and cross-lingual transfer ability. FRS and KTS measure the effectiveness of cross-lingual knowledge transfer, and X-FaKT Score integrates factual recall with transferability to provide a robust measure of multilingual generalization. These metrics offer a more nuanced evaluation than a simple error rate, allowing for a deeper understanding of cross-lingual generalization. In-Context Recall Machlab and Battle (2024) measures the general performance of the models in multilingual contexts. Inspired by Du et al. (2024) , we also study how factual knowledge of models affects their performance in handling in-context tasks in the multilingual setting ( Counterfactual Context Adherence ). For this, we design a dataset where factual knowledge conflicts with in-context instructions. Our experiments reveal that while LMs often retrieve factual information correctly in the language associated with the fact, they struggle to transfer this knowledge to other languages. We also found that the size of the LLM plays an important role in factuality and knowledge transferability. For example, the combined performance of LLama-3-70B in factuality and knowledge transfer across languages is markedly ( 152% \uparrow in X-FaKT Score) better than Llama-3.2-1B. In addition, there is a marked difference in these tasks when queries are asked in high-resource languages ( 46% \uparrow in X-FaKT Score) as compared to the case with low resources. This finding exposes a critical limitation in current language models and their approach to multilingual knowledge integration. Our findings also reveal an interesting trade-off: LMs with stronger factual recall often struggle with counterfactual adherence, highlighting a key limitation in balancing factual memory and contextual reasoning. In our experiments, we observed that the factual knowledge of LMs could skew their judgments, leading to inaccurate evaluations. One has to be very careful when designing the prompt and using LM as an evaluator. We highlight the importance of controlling the evaluator’s factual knowledge to ensure consistent and effective evaluation. 2 Related Work Multilingual Transformers. Early work by Petroni et al. (2019) explored whether LMs can store factual knowledge about entities, setting the stage for later investigations into multilingual LMs. Notable multilingual models such as mBERT Devlin et al. (2019) , XLM-R Conneau et al. (2020) , mT5 Xue et al. (2021) , and BLOOM Workshop et al. (2023) have demonstrated varying levels of performance across different languages. These models, trained on diverse multilingual corpora, show that LMs exhibit language-dependent capabilities in factual recall. Research has highlighted systematic biases in factual retrieval across languages Artetxe et al. (2020) ; Liu et al. (2020) ; Kassner et al. (2021) , which is a key challenge in multilingual LMs. While multilingual QA benchmarks like XQuAD Artetxe et al. (2020) , MLQA Lewis et al. (2020) , and TyDiQA Clark et al. (2020) assess factual consistency, they do not directly measure the transfer of knowledge between languages. Recent work by Wang et al. (2024) raised questions about LMs’ ability to recall factual knowledge in reasoning tasks, while Fierro et al. (2025) emphasized the need for more robust methodologies for evaluating knowledge in multilingual LMs. Our study builds on these insights by introducing a benchmark specifically designed to assess cross-lingual

Read the complete text