Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMs

Shmuel Berman, Yuval Kansal, Lydia Liu Princeton University, USA {shmuel, yuvalkansal, ltliu}@princeton.edu · 2025

Read Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMs by Shmuel Berman, Yuval Kansal, Lydia Liu Princeton University, USA {shmuel, yuvalkansal, ltliu}@princeton.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03051.txt.

Introduction

Kansal, Berman, and Liu test how well a leading model family answers school-level factual questions across languages in a 2025 paper. Correctness is a basic need for education tools. Strong English skill often hides weak performance elsewhere.

Questions suited to middle and high school students were posed to the Llama model family. In non-English languages, answers carried more extra material and less truth. Rare languages suffered most, so existing bias grew worse. The authors frame the gap as a fairness and safety issue for global classrooms. Their checks offer a template for auditing multilingual systems. The title's point is plain: facts should not depend on the language asked. Schools need proof before adoption.

Catalog record

CREATORShmuel Berman, Yuval Kansal, Lydia Liu Princeton University, USA {shmuel, yuvalkansal, ltliu}@princeton.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03051

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMs Shmuel Berman, Yuval Kansal, Lydia Liu Princeton University, USA {shmuel, yuvalkansal, ltliu}@princeton.edu Abstract Factuality is a necessary precursor to useful educational tools. As adoption of Large Language Models (LLMs) in education continues of grow, ensuring correctness in all settings is paramount. Despite their strong English capabilities, LLM performance in other languages is largely untested. In this work, we evaluate the correctness of the Llama3.1 family of models in answering factual questions appropriate for middle and high school students. We demonstrate that LLMs not only provide extraneous and less truthful information, but also exacerbate existing biases against rare languages. keywords AI in Education, LLMs, Question Answering, Factuality Introduction LLMs have shown exciting human-like capabilities in tasks such as text generation and question answering. Tools such as ChatGPT and LLama are being used in educational learning contexts and are employed directly by students without expert guidance. As they become ubiquitous learning tools, it is vital to ensure they reduce rather than reinforce existing inequities. Equitable access to education is essential for the betterment of society. The inconsistencies in LLMs for providing information to simple questions across multiple languages are concerning. In this study, we assess the factuality and consistency of LLM responses to middle and high school level questions in non-English languages. About 55% of the content on the internet is in English Wikipedia 2024 . All other languages make up a fraction of the content, with some accounting for only 0.1-0.2% of the internet websites. It has been shown that a LLM’s effectiveness is heavily dependent on the distribution and quality of training data Xie et al. 2024 . LLMs trained on poorly distributed data can undergo model collapse, gradually degenerate and produce increasingly homogeneous outputs Chen et al. 2024 . Recent studies have shown that “dominant” languages exhibit superior performance due to their substantial representation in training data, while low-resource languages demonstrate significantly degraded capabilities Xie et al. 2024 ; Ponti et al. 2019 . Insufficient training data in a language can result in the model losing valuable information about that language Chen et al. 2024 ; Chenxi Whitehouse 2023 . While prior work has evaluated language fluency and translation quality of LLMs Xie et al. 2024 ; Ponti et al. 2019 , our aim is to measure their factuality in different languages, especially in educational settings. We present three initial contributions in evaluating LLM factuality: (1) A benchmark of 54 factual questions and corresponding keyword-level answers spanning various subjects (2) A manual evaluation (IRB approved) of translated answer quality conducted by bilingual individuals (3) A statistical evaluation of several LLMs’ responses across languages which shows poorer performance across several metrics in non-English languages. Related Work Prior research has shown that LLMs tend to perform better in English and French compared to Bengali, Georgian, Nepali, and Maithili Dong et al. 2024 . The paper highlights safety alignment issues, the discrepancy in multilingual research, and proposes a new safety multilingual benchmark.“Factcheck-Bench” introduces a similar English-only dataset to evaluate LLM factual accuracy and proposes a pipeline to get better results Wang et al. 2024 These datasets limit their evaluation to factuality and do not take into account extraneous information. Researchers at Georgia Tech found that LLMs gave richer and higher-quality responses in English than in other languages Jin et al. 2023 . Wang et al. showed that ChatGPT performs much better at translating English to German than to Chinese Wang et al. 2023 . Li et al. showed that there is a direct relationship between a specific LLM’s performance in a language and the percentage of data belonging that language in the training corpus Li et al. 2024 . Zhu et al. showcased inconsistency across non-English languages and successfully boosted LLM performance in instruction following and translation through multilingual instruction tuning Zhu et al. 2023 . These studies quantify translation quality and contribute valuable benchmarks but do not discuss the factual information content. Methodology To assess LLM factuality, we compiled a list of 54 questions across different subjects covering middle and high school curriculum. Each question was designed to be answerable in a single sentence and is accompanied by manually curated target words essential for a correct response. For example, for the question “What is photosynthesis?”, target words included “sunlight”,“carbon dioxide”, and “glucose”. These questions were presented to Llama 3.1 (8B & 70B). The evaluation was conducted in 12 languages—Japanese, Arabic, French, Chinese, Persian, Hebrew, Hindi, Nepali, Haitian Creole, Tulu, Māori, and English. All responses were machine-translated back into English using the Google Translate API for analysis. While machine translation introduces potential limitations, prior research has demonstrated that Google Translate achieves adequate accuracy for tasks requiring factual content. Taira et al. assessed Google Translate’s performance and reported mean Likert scale adequacy scores exceeding 4 across languages Taira et al. 2021 . Our study focuses on factual accuracy, not fluency. Google Translate is sufficient for this purpose given that literal translations often suffice to convey essential information. For the manual analysis, bilingual evaluators assessed each response on a multidimensional rubric: counting the presence of each target keyword, factual accuracy (1-3 scale), and extraneous information (1-5 scale). For our automated analysis, we search for the presence of target words in both the original language and in the response translated to English, perform keyword extraction, and measure per-language response properties such as keyword repetition. We use KeyBERT Grootendorst 2020 to identify semantically important terms and extract them as keywords, and include metrics for the number of keywords and how many times they repeat. Our question bank and code can be found here . Results We conducted a manual evaluation of Llama-3.1-8B and 70B responses on English, Hindi, Hebrew, and present our averaged findings in the table below: Language Model # Speakers (millions) Keyword Coverage (%) Incorrectness 1 1 1 Incorrectness Score : Scale 1-3 — 1 if no incorrect information, 2 if some incorrect information, and 3 if clearly incorrect information. Extraneous Score 2 2 2 Extraneous Score : Scale 1-5 — 1 if no extraneous information, 5 if most of the answer is unnecessary. Hebrew LLaMA 8B 9 27.28 1.93 3.48 LLaMA 70B 9 69.25 1.44 2.11 Farsi LLaMA 70B 110 87.36 1.06 1.13 Hindi LLaMA 8B 345 51.17 1.50 2.13 LLaMA 70B 345 77.25 1.13 1.30 English LLaMA 8B 1500 94.44 1.04 1.35 LLaMA 70B 1500 97.48 1.09 1.26 Table 1: Manual Analysis Results for LLaMA 8B and LLaMA 70B. Keyword Coverage (higher is better), Incorrectness Score 1 1 1 Incorrectness Score : Scale 1-3 — 1 if no incorrect information, 2 if some incorrect information, and 3 if clearly incorrect information. , and Extraneous Score 2 2 2 Extraneous Score : Scale 1-5 — 1 if no extraneous information, 5 if most of the answer is unnecessary. (lower is better). To analyze the relationship between language representation and response quality, we present the Spearman Correlation coefficients for target word detection correlating the number of speakers of the language of each response with its features. We determine there is a statistically significant correlation between the presence of our hand-picked target words. For the larger model, there is also a strong correlation

Read the complete text