HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detection
Maryam Bala, Amina Imam Abubakar, Abdulhamid Abubakar, Abdulkadir Shehu Bichi, Hafsa Kabir Ahmad, Sani Abdullahi Sani, Idris Abdulmumin, Shamsuddeen Hassan Muhamad, Ibrahim Said Ahmad · 2025
Read HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detection by Maryam Bala, Amina Imam Abubakar, Abdulhamid Abubakar, Abdulkadir Shehu Bichi, Hafsa Kabir Ahmad, Sani Abdullahi Sani, Idris Abdulmumin, Shamsuddeen Hassan Muhamad, Ibrahim Said Ahmad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19650.txt.
Introduction
Maryam Bala and co-authors report their entry in a shared task on spotting hallucinated spans in AI text, in a 2025 paper. The MU-SHROOM task asks teams to mark the exact stretches where large language models invent facts, across outputs in 14 languages. The HausaNLP team aimed at a fine-grained, model-aware reading of the English data.
They combined a logic-check method, natural language inference, with a ModernBERT model fine-tuned on 400 synthetic examples. Scores were modest: the marked spans overlapped the truth by only 0.032 on the overlap index, while the model's confidence tracked real hallucinations with a correlation of 0.422. The team note that invented text hides in context, so its borders are hard to pin, and they frame the run as groundwork.
Catalog record
| CREATOR | Maryam Bala, Amina Imam Abubakar, Abdulhamid Abubakar, Abdulkadir Shehu Bichi, Hafsa Kabir Ahmad, Sani Abdullahi Sani, Idris Abdulmumin, Shamsuddeen Hassan Muhamad, Ibrahim Said Ahmad |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detection Maryam Bala, Amina Imam Abubakar, Abdulhamid Abubakar, Abdulkadir Shehu Bichi, Hafsa Kabir Ahmad, Sani Abdullahi Sani, Idris Abdulmumin, Shamsuddeen Hassan Muhamad, Ibrahim Said Ahmad Abstract This paper presents our findings of the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes, MU-SHROOM, which focuses on identifying hallucinations and related overgeneration errors in large language models (LLMs). The shared task involves detecting specific text spans that constitute hallucinations in the outputs generated by LLMs in 14 languages. To address this task, we aim to provide a nuanced, model-aware understanding of hallucination occurrences and severity in English. We used natural language inference and fine-tuned a ModernBERT model using a synthetic dataset of 400 samples, achieving an Intersection over Union (IoU) score of 0.032 and a correlation score of 0.422. These results indicate a moderately positive correlation between the model’s confidence scores and the actual presence of hallucinations. The IoU score indicates that our model has a relatively low overlap between the predicted hallucination span and the truth annotation. The performance is unsurprising, given the intricate nature of hallucination detection. Hallucinations often manifest subtly, relying on context, making pinpointing their exact boundaries formidable. 1 Introduction Despite the advancements in Natural Language Processing (NLP) and the development of Natural Language Generation (NLG) models, their limitations and potential risks have gained increased attention. A significant issue is that NLG models often produce unfaithful text relative to the source input, a phenomenon known as "hallucination" Koehn and Knowles 2017a ; Rohrbach et al. 2018 ; Maynez et al. 2020a . Hallucinations in NLG models often result in outputs that, while fluent, lack accuracy. This issue arises because existing evaluation metrics prioritize fluency over correctness, ultimately diminishing system performance and failing to meet user expectations in practical applications Mickus et al. 2024 . For example, Dopierre et al. 2021 illustrate this phenomenon by attempting to paraphrase the statement "I am not sure where my phone is," which leads to the hallucinated output: "How can I find the location of any Android mobile." Hallucination in NLG is concerning due to its impact on performance and safety in applications like medicine, where hallucinatory summaries or machine-translated instructions can pose risks to patient diagnosis Ji et al. 2023 . Similarly, hallucinations can lead to privacy violations by generating sensitive information not present in the source input Carlini et al. 2021 . To address this challenge, “SemEval-2025 Task 3 – Mu-SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes” Vázquez et al. 2025 was introduced. The shared task involves detecting specific text spans that constitute hallucinations in outputs generated by LLMs. Participants compute the probability of each character being marked as a hallucination using the LLM output consisting of a character string, tokens, and logits, thus enabling a fine-grained hallucination detection. 2 Background Numerous efforts are provided to address hallucination across various NLG tasks. Analyzing hallucinatory content and its relationships in different tasks could enhance our understanding and unify efforts across NLG fields Ji et al. 2023 . While most existing studies focus on specific tasks like abstractive summarization Huang et al. 2021 ; Maynez et al. 2020b and machine translation Lee et al. 2018 , the study of Ji et al. 2023 offer a comprehensive analysis on the phenomenon of hallucination in abstractive summarization, dialogue generation, generative question answering, data- to-text generation, and machine translation. Parikh et al. 2020 argue that hallucination problem occurs when there is very little divergence in dataset and encoder with a defective comprehension ability could influence the degree of hallucination. Similarly, Koehn and Knowles 2017b show that training and modeling choices of neural models have influence for hallucination. While large pre-trained models used for downstream NLG tasks are powerful in providing generalizability and coverage, they however prioritize parametric knowledge over the provided input and can result in hallucination of excess information in the output Longpre et al. 2021 . Recent efforts to address hallucination include the Shared Task on Hallucinations and Related Observable Overgeneration Mistakes (SHROOM), which focused on binary classification for task-specific English language models Mickus et al. 2024 . Maksimov et al. 2024 ; Arzt et al. 2024 ; Rahimi et al. 2024 identify cases of fluent overgeneration hallucinations in model-aware and model-agnostic settings. They detect grammatically sound outputs which contain incorrect or unsupported semantic information. Building on this task instead of focusing on model-agnostic and model-aware tracks, this year’s task focuses on the multilingual aspect. Therefore, all data-points in this year’s task are model-aware. We present our Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes (Mu-SHROOM) which aims to detect hallucinations in a multilingual context, providing a more nuanced understanding of their occurrence and severity. MU-SHROOM is a SemEval-2025 shared task that focuses on detecting hallucinations and overgeneration errors in AI-generated text, a crucial challenge in improving the reliability of LLMs Vázquez et al. 2025 . Hallucinations occur when LLMs produces fluent but false or unsupported content, while overgeneration mistakes involve excessive, often misleading text Ji et al. 2023 . This task is important as language models become increasingly prevalent in various applications, where factual accuracy and reliability are essential for maintaining user trust and system integrity. Therefore, tackling these issues is essential for ensuring AI-generated text remains trustworthy and useful across various applications Maksimov et al. 2024 . The task consists of participants detecting spans of text corresponding to hallucinations and determine which parts of the given text produced by LLMs constitute hallucinations. Annotated dataset are provided, allowing researchers to develop and benchmark models for identifying these issues in different linguistic contexts. Given the LLM output as a string of characters, a list of tokens, and a list of logits, participants calculate the probability that each character is marked as a hallucination and thus provide a fine- grained hallucination detection. Similarly, the task is held in multilingual and multi-model context as data are produced by a variety of public-weights LLM in multiple languages which includes: Arabic (Modern standard), Chinese (Mandarin), English, Finnish, French, German, Hindi, Italian, Spanish, and Swedish, along with three surprise test languages. 3 System Overview One of the primary challenges in this task was the lack of labeled training data. To address this, we created a synthetic training dataset using ChatGPT, adhering to official annotation guidelines. For the language selection, we used English language only and this is due to time constraint. Secondly, English language offers an extensive resource for NLP model development and evaluation. Similarly, focusing on a single language allowed us to develop deeper linguistic pattern recognition for hallucination detection. For model selection, we fine-tuned ModernBERT Warner et al. 2024 , an advanced variant of BERT Devlin et al. 2019 that offers significant improvements in handling long-context inputs, computational efficiency, and robustness in token classification tasks. This choice aligns with recent studies highlighting the benefits of long-context language models in detecting nuanced text errors Sahitaj et al. 2025 . For a precise understanding of hallucination detection, we used the model-aware method. This method is based on analysing internal data of LLM during inference. One of the possible approaches is the analysis of the outputs of the hidden layers of the transformer. Using