Poly-FEVER : A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models

Hanzhi Zhang, Sumera Anjum, Heng Fan, Weijian Zheng, Yan Huang, Yunhe Feng · 2025

Read Poly-FEVER : A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models by Hanzhi Zhang, Sumera Anjum, Heng Fan, Weijian Zheng, Yan Huang, Yunhe Feng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16541.txt.

Introduction

Hanzhi Zhang and five co-authors introduce Poly-FEVER in this two thousand twenty-five paper. Large language models often state false claims with confidence, a flaw called hallucination. Tests that catch this flaw focus mostly on English. That leaves model behavior in other languages unclear. Poly-FEVER is a fact-checking benchmark built to close that gap.

The set holds nearly seventy-eight thousand labeled claims across eleven languages. The claims come from three existing fact-check sources, on general facts, climate, and science. The authors use it to test models such as ChatGPT and the LLaMA series. Their analysis shows how topic mix shifts model scores across languages. The benchmark gives builders a shared yardstick for truthfulness beyond English.

Catalog record

CREATORHanzhi Zhang, Sumera Anjum, Heng Fan, Weijian Zheng, Yan Huang, Yunhe Feng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16541

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Poly-FEVER : A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models Hanzhi Zhang, Sumera Anjum, Heng Fan, Weijian Zheng, Yan Huang, Yunhe Feng Abstract . Hallucinations in generative AI, particularly in Large Language Models (LLMs), pose a significant challenge to the reliability of multilingual applications. Existing benchmarks for hallucination detection focus primarily on English and a few widely spoken languages, lacking the breadth to assess inconsistencies in model performance across diverse linguistic contexts. To address this gap, we introduce Poly-FEVER , a large-scale multilingual fact verification benchmark specifically designed for evaluating hallucination detection in LLMs. Poly-FEVER comprises 77,973 labeled factual claims spanning 11 languages, sourced from FEVER, Climate-FEVER, and SciFact. It provides the first large-scale dataset tailored for analyzing hallucination patterns across languages, enabling systematic evaluation of LLMs such as ChatGPT and the LLaMA series. Our analysis reveals how topic distribution and web resource availability influence hallucination frequency, uncovering language-specific biases that impact model accuracy. By offering a multilingual benchmark for fact verification, Poly-FEVER facilitates cross-linguistic comparisons of hallucination detection and contributes to the development of more reliable, language-inclusive AI systems. The dataset is publicly available to advance research in responsible AI, fact-checking methodologies, and multilingual NLP, promoting greater transparency and robustness in LLM performance. The proposed Poly-FEVER is available at: https://huggingface.co/datasets/HanzhiZhang/Poly-FEVER . 1. Introduction LLMs, such as those in the GPT family, have exhibited remarkable proficiency across diverse domains including education, healthcare, and legal affairs. In these applications, the accuracy and factual integrity of the content generated by LLMs are critical, particularly in areas requiring precise guidance, like medical and legal advice. Despite their advancements, mainstream LLMs predominantly utilize corpora that are imbalanced in terms of demographic groups ( Shah et al., 2019 ; Li et al., 2023 ) . Language, an important facet of demographic backgrounds, remains relatively underinvestigated in the context of detecting hallucinatory content in LLMs, particularly from the perspective of fairness and equitable usability. Previous research has predominantly focused on hallucinations in LLMs within widely spoken languages, such as English ( Yao et al., 2023 ) , Chinese ( Cheng et al., 2023 ) , and German ( Sennrich et al., 2023 ) . This focus has led to a thorough understanding of hallucinatory outputs’ mechanisms and the development of mitigation strategies. One approach involves prompt engineering, which includes retrieval augmentation to ground content in external evidence ( Lewis et al., 2020 ) , feedback loops for refining responses, and prompt tuning to adjust prompts during fine-tuning for desired behaviors. Another strategy is model development ( Tonmoy et al., 2024 ) , focusing on creating models inherently less prone to hallucinating through architectural changes, novel loss functions, and supervised fine-tuning using human-labeled data. Despite the critical insights gained, the focus on major languages has marginalized the experiences and challenges of LLMs trained on or applied to less common languages. Moreover, these investigations often employ differing datasets, leading to an absence of a systematic approach to assessing hallucinations across languages with uniform input. The complexity of this issue is multifold. First, there is a scarcity of appropriate datasets for cross-linguistic studies. Second, accurately detecting hallucinations on a large scale can be challenging, particularly in topics intertwined with local cultural and linguistic contexts. Third, the generation of apparently plausible yet inaccurate content manifests due to biased training and fine-tuning referenced resources on languages of marginal transmission. Nonetheless, ensuring equitable performance of LLMs across languages, especially those that are underrepresented, is crucial from the perspective of hallucination consistency. This paper examines hallucination detection in LLMs across multiple languages by expanding fact verification into a multilingual benchmark, Poly-FEVER . Fact verification and hallucination detection involve different objectives. Fact verification determines whether a claim aligns with established knowledge sources ( Murayama, 2021 ; Zhu et al., 2021 ) . This process requires external evidence to confirm or refute statements. Hallucination detection, however, identifies false or misleading claims without relying on external references. An LLM that fabricates information may generate statements that appear factual but contain errors, inconsistencies, or contradictions. Evaluating hallucinations requires analyzing the model’s outputs to detect inaccuracies, regardless of whether external knowledge exists. In multilingual contexts, hallucination detection becomes more complex because models process languages with different data distributions, resource availability, and linguistic structures. Poly-FEVER addresses this challenge by enabling a structured evaluation of hallucinations across multiple languages, exposing discrepancies in model performance and assessing factual consistency without dependence on external sources. We summarize the key contributions as follows: (1) We introduce Poly-FEVER , an extensive, publicly available dataset tailored for multilingual fact extraction and verification. It covers 11 languages and includes 77,973 fact claims on various topics, designed for hallucination detection tasks. Poly-FEVER is available at: https://huggingface.co/datasets/HanzhiZhang/Poly-FEVER . (2) We analyze hallucination detection capabilities in advanced language models of ChatGPT, LLaMA-2 series (7B, 13B, and 70B), and LLaMA-3.1 8B, using Poly-FEVER with both language-wise and classification prompts. (3) We investigate the reasons behind hallucinations on a multilingual scale, employing Latent Dirichlet Allocation (LDA) for topic distribution analysis and automated web searches to assess resource imbalances. 2. Related Work LLMs generate hallucinations—outputs that contradict known facts or fabricate non-existent information, posing challenges for reliable AI applications. Huang et al. (2023) categorize hallucinations into two types: intrinsic and extrinsic. Intrinsic hallucinations occur when a model produces self-contradictions or logical inconsistencies within its own response, while extrinsic hallucinations involve factually incorrect statements that do not align with established knowledge sources. Evaluating hallucinations requires a systematic comparison between generated content and verified ground-truth data or an analysis of internal inconsistencies within model outputs. Effective hallucination detection plays a crucial role in ensuring the credibility of generative AI systems, particularly in high-stakes applications such as healthcare, legal services, and scientific research. Existing evaluation metrics measure hallucination frequency by assessing entity correctness, factual consistency, and contradiction detection. Early methods relied on n-gram overlap (e.g., ROUGE, PARENT-T) ( Lin, 2004 ; Wang et al., 2020 ) , but these metrics fail to capture factual accuracy. Entity-based hallucination metrics ( Nan et al., 2021 ) and relation-based fact extraction models ( Goodrich et al., 2019 ) provide more structured approaches. However, these methods primarily target English-language outputs, limiting their applicability to multilingual hallucination analysis. To assess hallucination detection, researchers rely on fact-checking datasets such as FEVER ( Thorne et al., 2018 ) , Climate-FEVER ( Diggelmann et al., 2020 ) , and SciFact ( Wadden et al., 2020 ) . These datasets evaluate model responses by comparing claims against external sources. However, they primarily focus on fact verification, not hallucination detection, since they assume that relevant evidence is always available. Current benchmarks also exhibit strong language biases. Most fact-checking datasets contain English-only claims, restricting their ability to evaluate how LLMs handle fact verification across different linguistic contexts. While some efforts attempt multilingual fact-checking ( Gupta and Srikumar, 2021 ) , these datasets typically do not provide identical claims across languages, making it difficult to access cross-lingual hallucination patterns. Most research on multilingual hallucinations focuses on machine translation errors, where LLMs generate hallucinated translations ( Pfeiffer et al., 2023 ) . However, translation-based hallucination benchmarks do not evaluate self-contained factual inconsistencies within LLM outputs. LLMs trained on English-centric corpora often exhibit language-specific hallucination

Read the complete text