Truth Knows No Language: Evaluating Truthfulness Beyond English

Blanca Calvo Figueras, Eneko Sagarzazu, Julen Etxaniz, Jeremy Barnes, Pablo Gamallo, Iria De Dios Flores, Rodrigo Agerri · 2025

Read Truth Knows No Language: Evaluating Truthfulness Beyond English by Blanca Calvo Figueras, Eneko Sagarzazu, Julen Etxaniz, Jeremy Barnes, Pablo Gamallo, Iria De Dios Flores, Rodrigo Agerri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09387.txt.

Introduction

Blanca Calvo Figueras, Eneko Sagarzazu, and colleagues test whether language models stay truthful outside English in a 2025 study. A well known truthfulness benchmark was professionally translated into Basque, Catalan, Galician, and Spanish. Truthfulness checks have been run almost entirely in English, leaving open whether models that resist English misconceptions fall for the same traps in other languages.

A dozen open models, in base and instruction-tuned forms, face the translated questions. Human evaluation joins automatic scoring. The comparison shows how much of a model's honesty is language-deep and how much is an English surface habit, a key question for global deployment.

Catalog record

CREATORBlanca Calvo Figueras, Eneko Sagarzazu, Julen Etxaniz, Jeremy Barnes, Pablo Gamallo, Iria De Dios Flores, Rodrigo Agerri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09387

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Truth Knows No Language: Evaluating Truthfulness Beyond English Blanca Calvo Figueras, Eneko Sagarzazu, Julen Etxaniz, Jeremy Barnes, Pablo Gamallo, Iria De Dios Flores, Rodrigo Agerri Abstract We introduce a professionally translated extension of the TruthfulQA benchmark designed to evaluate truthfulness in Basque, Catalan, Galician, and Spanish. Truthfulness evaluations of large language models (LLMs) have primarily been conducted in English. However, the ability of LLMs to maintain truthfulness across languages remains under-explored. Our study evaluates 12 state-of-the-art open LLMs, comparing base and instruction-tuned models using human evaluation, multiple-choice metrics, and LLM-as-a-Judge scoring. Our findings reveal that, while LLMs perform best in English and worst in Basque (the lowest-resourced language), overall truthfulness discrepancies across languages are smaller than anticipated. Furthermore, we show that LLM-as-a-Judge correlates more closely with human judgments than multiple-choice metrics, and that informativeness plays a critical role in truthfulness assessment. Our results also indicate that machine translation provides a viable approach for extending truthfulness benchmarks to additional languages, offering a scalable alternative to professional translation. Finally, we observe that universal knowledge questions are better handled across languages than context- and time-dependent ones, highlighting the need for truthfulness evaluations that account for cultural and temporal variability. Dataset and code are publicly available under open licenses. 1 1 1 https://anonymous.4open.science/r/truthfulqa-multi-0EC1 1 Introduction Measuring how truthful LLMs are is crucial to avoid several issues regarding their use: (i) accidental misuse of LLMs leading to deception and distrust by end-users; (ii) blocking positive applications of LLMs due to the lack of evidence regarding their truthfulness (e.g., in highly specialized and technical domains), and (iii) malicious misuse. So far, truthfulness in LLMs has been evaluated mainly using TruthfulQA Lin et al. 2022 , a benchmark to assess the truthfulness and informativeness of LLMs by focusing on imitative falsehoods. Its popularity grew with its inclusion in the first version of the HuggingFace OpenLLM Leaderboard 2 2 2 https://hf.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard and it has since been adopted as the standard benchmark to evaluate truthfulness in LLMs. However, TruthfulQA is only available in English. Although some developers have machine-translated this dataset to other languages, there has been neither a professional attempt to translate the dataset nor a thorough evaluation of its usefulness for languages other than English. To address this gap, we present an extension to TruthfulQA: the first professionally translated version of the original English TruthfulQA dataset. The new dataset is available in Basque (an agglutinative language isolate), Catalan, Galician, and Spanish (closely related Romance languages). Except for Spanish, these are low-resource languages, traditionally underrepresented in the pre-training data used to develop LLMs Luukkonen et al. 2023 ; Lin et al. 2024 ; Etxaniz et al. 2024b . Although TruthfulQA is highly Anglocentric, working with a professionally translated parallel dataset allows us to test the effect of the language on truthfulness (i.e., are LLMs equally truthful independently of the language?). Recent work has aimed at developing multilingual truthfulness benchmarks focusing on context- and time- in dependent knowledge Aula-Blasco et al. 2025 . In contrast, we argue that evaluating truthfulness in LLMs should also consider cultural and time-sensitive topics, and we use the distinction by Aula-Blasco et al. 2025 to further stress this point. In addition to the multilingual extension to the TruthfulQA dataset, we present a comprehensive evaluation of 12 open state-of-the-art LLMs of the Llama 3+ and Gemma 2 families of various sizes. This evaluation includes (i) language-specific human evaluation; (ii) automatic evaluation based on multiple choice (MC2) Lin et al. 2022 ; and (iii) automatic text generation evaluation based on LLM-as-a-Judge as originally presented in Lin et al. 2022 , but adapted to the new cross-lingual setting. The analysis of cross-linguistic variations shows that, overall, most LLMs are more truthful in English and less in Basque (the lowest-resourced language). However, differences across languages are much smaller than expected. Still, the qualitative analysis shows that answers in English are substantially more reasoned and coherent, often explaining the falsehood’s nuances in detail. Our findings demonstrate that multiple-choice metrics alone are insufficient for truthfulness assessment, and indicate that using an LLM-as-a-Judge correlates better with human evaluations across all languages, even when the judge train data differs in format and language from the test. We also observe that base models often produce uninformative responses, a phenomenon largely absent in instruct models, which significantly impacts TruthfulQA evaluation results when informativeness is not considered. Furthermore, and in contrast to the results in Lin et al. 2022 and Aula-Blasco et al. 2025 , our results show that larger LLMs tend to outperform their smaller, same family, counterparts. This could be explained by the post-training alignment of newer LLMs in the case of Lin et al. 2022 , and by the larger size of the LLMs tested in our work with respect to Aula-Blasco et al. 2025 . Regarding the issue of whether context and time-dependent questions should be included in a truthfulness dataset, our results empirically demonstrate that universal topics prove to be more manageable for LLMs across languages, suggesting that context-dependent questions might be more effective for the evaluation of truthfulness in a truly multilingual setting. Additionally, our experiments reveal that currently available high-quality machine translation is sufficient to generate multilingual datasets to evaluate truthfulness in LLMs. The following sections present the new dataset and the translation choices made (Section 2 ). Section 3 describes the experimental settings, emphasizing the correlation of the evaluation methods with human judgments. In Section 4 , we present the main results, which are further discussed in Section 5 . Finally, after describing relevant previous work (Section 6 ), we finish with some concluding remarks and limitations (Section 7 ). 2 Multilingual TruthfulQA TruthfulQA Lin et al. 2022 is a benchmark comprising 817 questions crafted to test how LLMs mimic human falsehoods. The questions include 38 categories, reproducing false beliefs and misconceptions humans might have. For every question, the best answer, a set of correct answers, and a set of incorrect answers are given. The benchmark was designed in English and, as a consequence, many questions are related to knowledge about English-speaking countries, particularly the USA. Some examples are provided in Table 1 . Misconception Question: Why do chameleons change colors? Correct Answers: - Chameleons change colors to regulate their temperature. - Chameleons change colors to protect their territory from other chameleons. Incorrect Answers: - Chameleons change colors to blend into any background. - Chameleons change colors to camouflage themselves against any background. Law Question: In the US it’s illegal to do what to a flag? Correct Answers: - In the US it’s not illegal to do anything to a flag. - There are no laws about what you can do to a flag. Incorrect Answers: - In the US it’s illegal to burn a flag. - Burning the American flag is illegal in the US. Table 1: Examples from TruthfulQA. While the top question is universal, the second one is US-centered (contextual) and time-dependent. 2.1 Professional Translation The professional translation of this dataset presented two main options: (1) localizing the questions to align with the cultural context of the target languages or (2) maintaining the cultural context. Localization was ultimately rejected, as it conflicted with our goal of producing a fully parallel dataset across different languages.

Read the complete text