HalluVerse25 : Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations

Samir Abdaljalil, Hasan Kurban, Erchin Serpedin · 2025

Read HalluVerse25 : Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations by Samir Abdaljalil, Hasan Kurban, Erchin Serpedin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07833.txt.

Introduction

Samir Abdaljalil, Hasan Kurban, and Erchin Serpedin build a test set for catching made-up facts in language model output in a 2025 paper. Such errors, called hallucinations, come in fine grades: a wrong name, a wrong link between facts, or a whole wrong sentence. Older test sets rarely capture that detail, and fewer still work across languages.

Their set, HalluVerse25, sorts fine-grained hallucinations in English, Arabic, and Turkish. A language model planted falsehoods into true life-story sentences, and human checkers then cleaned the result. The team ran several models on the set to grade how well each spots planted errors. Their findings compare closed and open models across the three languages.

Catalog record

CREATORSamir Abdaljalil, Hasan Kurban, Erchin Serpedin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HalluVerse25 : Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations Samir Abdaljalil, Hasan Kurban, Erchin Serpedin Abstract \setcode utf8 Abstract Large Language Models (LLMs) are increasingly used in various contexts, yet remain prone to generating non-factual content, commonly referred to as “hallucinations”. The literature categorizes hallucinations into several types, including entity-level, relation-level, and sentence-level hallucinations. However, existing hallucination datasets often fail to capture fine-grained hallucinations in multilingual settings. In this work, we introduce HalluVerse25 , a multilingual LLM hallucination dataset that categorizes fine-grained hallucinations in English, Arabic, and Turkish. Our dataset construction pipeline uses an LLM to inject hallucinations into factual biographical sentences, followed by a rigorous human annotation process to ensure data quality. We evaluate several LLMs on HalluVerse25 , providing valuable insights into how proprietary models perform in detecting LLM-generated hallucinations across different contexts. 1 Introduction Large Language Models (LLMs) have demonstrated promising results in various natural language processing (NLP) tasks, including summarization Liu et al. 2024 ; Ramprasad et al. 2024 , machine translation Zhu et al. 2024 , question answering Kamalloo et al. 2023 , and many more. Despite this, the phenomenon of producing fabricated and nonfactual content, known as "hallucinations”, has also been on the rise Banerjee et al. 2024 . Hallucinations can lead to the spread of misinformation, as they can sometimes appear to be plausible, particularly to individuals who are unfamiliar with the subject matter being generated Berberette et al. 2024 . As a result, analyzing and detecting hallucinations has become a major focus in the literature. Several datasets have been developed to understand and analyze hallucinations generated by LLMs Mubarak et al. 2024 ; Li et al. 2023 ; Manakul et al. 2023 ; Mishra et al. 2024 . Existing datasets, however, mostly consist of monolingual data, primarily in English, and often present hallucination in a binary classification setting. Such datasets lack the fine-grained annotation necessary for a deeper understanding of hallucination types. To fill this gap, we introduce HalluVerse25 , a fine-grained multilingual LLM hallucination dataset. Our main contributions are as follows: • We introduce the first multilingual fine-grained LLM hallucination dataset. The dataset spans across three languages, namely English, Arabic, and Turkish, with 1310, 828, and 978 data samples, respectively. This dataset is made publicly available to advance research in the field 1 1 1 Available Upon Request . • We detail a comprehensive and reproducible dataset creation and annotation protocol. • We perform a detailed analysis of the performance of proprietary LLMs on detecting fine-grained hallucinations in multiple settings. The remainder of the paper is organized as follows: Section 2 discusses relevant literature on hallucination datasets and benchmarks. Section 3 details the dataset construction methodology. Section 4 presents a comprehensive analysis of the dataset. Section 5 discusses experimental results of several LLMs on the dataset. Finally, Section 6 concludes the paper and outlines future directions. 2 Related Work Hallucination in LLMs. Hallucination in LLMs continues to be a pertinent issue with its diverse real-world applications Li et al. 2024 . An increasing proportion of the literature explores this phenomenon Huang et al. 2024 ; Ji et al. 2023 . A portion of the literature focuses on the underlying causes of hallucination McKenna et al. 2023 ; Das et al. 2022 , while other works propose several methodologies to detect and mitigate LLM hallucinations Chen et al. 2024a ; Chen et al. 2024b ; Varshney et al. 2023 . For instance, Chen et al. 2024a use the internal hidden states of the LLM to detect the presence of hallucinations using an Eigenscore metric. While significant progress has been made in understanding hallucination in LLMs, the development of comprehensive benchmarks and datasets remains crucial for evaluating and mitigating these phenomena. Hallucination Benchmarks & Datasets. To address the need for standardized and comprehensive evaluation, several benchmarks and datasets have been introduced in the literature, with the aim of capturing and quantifying hallucinations in various contexts. With 35,000 samples, HaluEval encompasses multiple tasks such as question answering, summarization, and open-ended generations in English Li et al. 2023 , while Mubarak et al. 2024 introduce Halwasa, a dataset comprised of 10,000 sentence-level hallucination samples in Arabic, extending the scope to non-English languages. Other smaller datasets, such as the Wikibio hallucination dataset Manakul et al. 2023 , were introduced in the literature. In addition to binary hallucination datasets, fine-grained hallucination types have also been defined and explored. Mishra et al. 2024 define several distinct types of factual errors including: • Entity-level: Errors where an incorrect entity in a statement can be replaced to make it accurate; • Relation: Errors caused by incorrect semantic relationships, such as verbs or prepositions; • Sentence-level: Errors where a large portion of the statement contradicts evidence. Using these hallucination types, the authors introduced FAVABENCH, a dataset consisting of 1,000 hallucinated query responses with fine-grained labels. Other fine-grained non-English hallucination datasets have also been recently presented in the literature, such as ANAH Ji et al. 2024 , a bilingual Chinese/English sentence-level annotated hallucination dataset that covers a variety of topics. Although progress has been made, a gap still exists in the literature for a self-contained, fine-grained multilingual hallucination dataset. Existing benchmarks mainly focus on individual languages, often limited to English, or lack the fine-grained annotations needed to explore and understand the nuances of hallucination across different languages. To fill this gap, we present HalluVerse25 , a fine-grained multilingual LLM hallucination dataset, which aims to provide in-depth insight into different types of hallucinations in multilingual contexts. 3 Constructing HalluVerse25 Figure 1: Pipeline of Dataset Construction, including extraction of biographical factual sentences, automatic injection of fine-grained hallucinations, and human annotation. The main objective behind the construction of this dataset is to understand what different types of hallucinations look like in multiple languages. The dataset includes the original factual sentences and their hallucinated versions, as well as the hallucination type associated with each edited sentence. The construction pipeline is illustrated in Fig. 1 . Furthermore, samples from the final dataset are shown in Table 1 . 3.1 Extraction of Biographical Data Wikipedia Links from Wikidata. For this work, we focus primarily on building a data set based on biographical data. The required data is extracted from Wikidata using a SPARQL query. Wikidata is a multilingual knowledge graph by the Wikimedia Foundation Abdaljalil and Mubarak 2024 . The query identifies 500 human entities from the Wikidata knowledge base (specifically those represented by the entity identifier wd:Q5 ). For each of these entities, the query retrieves the associated Wikipedia links in three languages: English, Arabic, and Turkish. The query uses Wikidata Property wdt: P2025 to obtain the respective links to the articles in these languages. The final output of the query provides a list of entities with their corresponding Wikipedia links in the specified languages, which are then used for further data extraction. The full SPARQL query used to extract this data is provided in Appendix A . Extracting Factual Sentences. The next step involved accessing the links and extracting the text from each page using BeautifulSoup 2 2 2 https://pypi.org/project/beautifulsoup4/ . First, we excluded pages that contained fewer than 20 sentences of text. This included pages that had minimal textual content, or did not contain any.

Read the complete text