CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation

Xu Liu and Guanyi Chen · 2025

Read CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation by Xu Liu and Guanyi Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11965.txt.

Introduction

Chatbots sometimes state falsehoods with full confidence, and shared tasks test how well software can spot those slips. Xu Liu and Guanyi Chen describe the Central China Normal University entry to SemEval-2025 Task 3 in their 2025 paper. The task, called Mu-SHROOM, asks systems to mark hallucinated spans in answers across fourteen languages.

Their system runs several large language models side by side, each acting as a separate annotator in a mock crowd. Each model mixes what it knows with facts pulled from outside sources before judging a span. Built on the open DeepSeek-V3 model, the system ranked first for Hindi and reached the top five in seven more languages. The paper also reports failed tries and lessons the pair took from the contest.

Catalog record

CREATORXu Liu and Guanyi Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11965

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation Xu Liu and Guanyi Chen Abstract We present the system developed by the Central China Normal University (CCNU) team for the Mu-SHROOM shared task, which focuses on identifying hallucinations in question-answering systems across 14 different languages. Our approach leverages multiple Large Language Models (LLMs) with distinct areas of expertise, employing them in parallel to annotate hallucinations, effectively simulating a crowdsourcing annotation process. Furthermore, each LLM-based annotator integrates both internal and external knowledge related to the input during the annotation process. Using the open-source LLM DeepSeek-V3, our system achieves the top ranking (#1) for Hindi data and secures a Top-5 position in seven other languages. In this paper, we also discuss unsuccessful approaches explored during our development process and share key insights gained from participating in this shared task. 1 Introduction Hallucinations refer to content in outputs that neither follow from the inputs nor are supported by known facts. In 2024, Mickus et al. 2024 organized a shared task on detecting hallucinations in machine translation, definition modelling, and paraphrasing systems. Building on this foundation and expanding to a new domain—question answering—SemEval-2025 Task 3 ( Vázquez et al. 2025 , Mu-SHROOM;) broadens the scope of hallucination detection. This task extends beyond English to cover 14 different languages and moves beyond binary classification (i.e., determining whether an item contains hallucinations) to pinpointing the exact location of hallucinations, as illustrated in Table 1 . Although Large Language Models (LLMs) inevitably produce hallucinations ( Xu et al. 2024 ) , they have also proven effective in detecting them: four of the six highest-scoring systems in the 2024 challenge leveraged state-of-the-art LLMs ( Mickus et al. 2024 ) . However, the new task setting introduced above presents two key challenges for these LLM-based solutions. First , Mu-SHROOM shifts the focus from hallucinations in generation systems, such as machine translation and paraphrasing, to hallucinations in question-answering (QA) systems. This shift alters the definition of hallucination. As discussed in Thomson and Reiter 2020 ; Dušek and Kasner 2020 ; Ji et al. 2023 ; van Deemter 2024 , hallucinations in generation systems refer to outputs that contradict the given inputs. In contrast, within QA, hallucinations pertain to outputs that contradict corresponding “facts”. Consequently, detecting hallucinations in a given QA pair requires a model first to determine what constitutes the relevant “facts”. Since these facts are not explicitly present in the input, the model must be capable of integrating knowledge from multiple sources. Question What did Petra van Staveren win a gold medal for? Answer Petra van Stoveren won a silver medal in the 2008 Summer Olympics in Beijing, China . Table 1: An example test item from Mu-SHROOM. The hallucinations are coloured in red. Figure 1: An overview of our hallucination annotation system. The blue whale represents LLM (i.e., DeepSeek). Second , the fine-grained hallucination annotation scheme in Mu-SHROOM increases the likelihood of annotation disagreements. Different annotators may label the same error in different ways. For example, consider the error “silver” in Table 1 : the term is incorrect because Petra van Stroveren won a gold medal in the Olympic Games. However, one annotator might highlight only the word “silver”, while another might annotate the entire noun phrase “a silver medal”. Such disagreements are natural, and Mu-SHROOM addresses them by employing multiple annotators and resolving inconsistencies through majority voting. This collaborative approach is difficult to replicate with a single LLM-based hallucination detector. Following the approach of the 2024 challenge winners, our solution employs LLMs with optimizations to address the two challenges discussed above. To tackle the first issue, our LLM-based hallucination detector retrieves relevant “facts” not only from its internal knowledge but also from external resources, thereby integrating both internal and external knowledge. To address the second issue, our solution mimics the crowdsourced annotation process by leveraging multiple LLMs, assigning them different roles, and having them annotate each QA pair in parallel before reaching a consensus through voting. Notably, our approach requires no fine-tuning or language-specific optimizations. Using an open-source LLM—DeepSeek-V3 ( Liu et al. 2024 ) —as the backbone, our solution achieved #1 ranking on Hindi data and placed in the Top 5 for Arabic, Basque, Catalan, Czech, English, Persian, and Spanish. 1 1 1 For German and French, we used GPT-4o, ranking #3 and #15, respectively. 2 The Mu-SHROOM Task The Mu-SHROOM task ( Vázquez et al. 2025 ) asks systems to annotate hallucinations in QA in 14 languages, including Arabic, Basque, Catalan, Chinese, Czech, English, Farsi, Finnish, French, German, Hindi, Italian, Spanish, and Swedish. The annotations contain: (1) Hard Labels , i.e., hallucinations in QA pairs as in Table 1 ; and (2) Soft Labels , i.e., probability of each token in the answer being a hallucination term. Mu-SHROOM evaluates each system using Intersection-over-Union (IoU) for hard labels and Spearman correlation (Cor) for soft labels. See Vázquez et al. 2025 for more details. 3 Methodology This section starts with explaining how we prompt LLMs to annotate hallucinations in QA systems, followed by how we make them leverage internal and external knowledge during annotation. Figure 1 provides an overview of our hallucination annotation system. 3.1 Prompting LLMs to Mark Hallucinations Figure 2: The main prompt in our system. The variables highlighted in yellow will be replaced with their corresponding desired values. As shown in Figure 2 , our prompt 2 2 2 For all languages, the prompt is always in English, with the only modification being the replacement of ‘lang’ with the name of the test language. begins by defining the task and the concept of hallucination to provide the LLM with a clearer understanding of the background. Notably, we refine the definition of hallucination in the context of QA by specifying that hallucinated content is characterized as “factually incorrect”, “nonsensical”, or “not supported by known facts”. We then incorporate a Chain-of-Thought (CoT), outlining the steps the LLM should follow to improve hallucination annotation. In this CoT, we first instruct the LLM to generate a reference answer based on the given question (see further discussion in Section 3.2 ). It then compares the provided answer with the generated reference answer to identify hallucinated content. We also ask the LLM to explain why the annotated terms are classified as hallucinations. Next, we present the LLM with an example, extracted from the first item in the development set. We also specify the expected input and output format. It is worth mentioning that rather than directly returning soft and hard labels—where hallucinations are represented as integer indices indicating their start and end positions—we instruct the LLM to mark hallucinated terms within the answer. This is achieved by having the LLM generate a revised version of the answer, where hallucinated terms are enclosed within \langle\langle ’ and \rangle\rangle ’. This approach bypasses the LLM’s limited ability to accurately count indices. Finally, we provide the LLM with the input QA pair along with external knowledge (see further discussion in Section 3.3 ). For each QA pair, we prompt the LLM 12 times and obtain 12 annotations. For each token in a given answer, we calculate the probability of it being hallucinated by computing

Read the complete text