MSA at SemEval-2025 Task 3: High Quality Weak Labeling and LLM Ensemble Verification for Multilingual Hallucination Detection

Baraa Hikal, Ahmed Nasreldin, Ali Hamdi · 2025

Read MSA at SemEval-2025 Task 3: High Quality Weak Labeling and LLM Ensemble Verification for Multilingual Hallucination Detection by Baraa Hikal, Ahmed Nasreldin, Ali Hamdi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20880.txt.

Introduction

Hikal, Nasreldin, and Hamdi built a fake-span spotter in twenty twenty-five for a shared task on made-up text in many tongues. Word models oft add false claims in their output, in any tongue they serve. Their plan has one model pull the suspect spans first.

Three more models then vote on each span by score odds. A fuzzy match step lines the spans up with the source text. It placed first in two tongues and near top in six more. The vote plan mimics how human marks were checked, they note.

Catalog record

CREATORBaraa Hikal, Ahmed Nasreldin, Ali Hamdi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20880

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MSA at SemEval-2025 Task 3: High Quality Weak Labeling and LLM Ensemble Verification for Multilingual Hallucination Detection Baraa Hikal, Ahmed Nasreldin, Ali Hamdi Abstract This paper describes our submission for SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes Vázquez et al. 2025 . The task involves detecting hallucinated spans in text generated by instruction-tuned Large Language Models (LLMs) across multiple languages. Our approach combines task-specific prompt engineering with an LLM ensemble verification mechanism, where a primary model extracts hallucination spans and three independent LLMs adjudicate their validity through probability-based voting. This framework simulates the human annotation workflow used in the shared task validation and test data. Additionally, fuzzy matching refines span alignment. Our system ranked 1 st in Arabic and Basque, 2 nd in German, Swedish, and Finnish, and 3 rd in Czech, Farsi, and French. 1 1 footnotetext: https://github.com/baraahekal/mu-shroom 1 Introduction Large Language Models (LLMs) are highly effective in generating text; however, they sometimes produce hallucinations—misleading content that is not properly grounded in the input data Huang et al. 2025 . Identifying these spans is essential for improving the reliability of LLM-generated outputs in translation, summarization, and conversational AI Alaharju 2024 . SemEval-2025 Task 3: Mu-SHROOM tackles this challenge by presenting a multilingual benchmark for detecting character-level hallucinations across multiple languages. The task involves detecting hallucinated spans in instruction-tuned LLM outputs, presenting challenges in language diversity, annotation consistency, and accurate span localization. Sriramanan et al. 2025 To tackle this challenge, our system utilizes a hybrid approach that integrates task-specific prompt engineering for weak label generation with an LLM ensemble verification mechanism Hikal et al. 2025 . Our methodology follows a multi-step adjudication process in which a primary LLM identifies hallucination spans, and three independent LLMs subsequently verify their validity through a probability-based voting mechanism Kang et al. 2024b . Additionally, we apply fuzzy matching techniques to improve the alignment of hallucination spans with ground truth annotations, thereby enhancing detection accuracy Chaudhuri et al. 2003 . By participating in this task, we gained insights into language-specific hallucination challenges and the strengths and limitations of LLM-based verification. Certain LLMs demonstrated closer alignment with human annotations, while hallucination patterns varied significantly, particularly in morphologically rich languages where annotation ambiguity was higher Abdelrahman 2024 . Our results indicate that ensemble verification and span refinement substantially improve hallucination detection, offering a robust approach for mitigating LLM hallucinations in multilingual settings. 2 Related Work Hallucination detection in Large Language Models (LLMs) has been studied in machine translation, text summarization, and conversational AI Ji et al. 2023 . Earlier approaches primarily relied on sentence-level classification, whereas recent research has transitioned to span-level detection for greater precision Joshi et al. 2020 . Self-consistency verification and knowledge-grounded approaches have improved hallucination identification, but many depend on external data, limiting their applicability in multilingual settings. Mehta et al. 2024 Multilingual NLP models struggle with hallucinations, especially in low-resource languages where confidence scores are unreliable Kang et al. 2024a . Morphologically rich languages introduce additional challenges due to intricate annotation inconsistencies Tsarfaty et al. 2013 . Prior work on translation-based verification has attempted to address this, but these approaches are ineffective in zero-shot scenarios Nie 2022 . Ensemble verification methods enhance detection accuracy by utilizing multiple models. Approaches such as multi-agent verification and cross-model adjudication have proven effective in assessing LLM outputs Liu and Wang 2024 . Our system expands on these approaches by integrating weak label generation with an ensemble verification pipeline, while also utilizing fuzzy matching to improve span alignment. Unlike previous methods that rely on single-model hallucination detection, our approach leverages an ensemble of LLMs for adjudication, reducing model bias and improving hallucination span refinement via fuzzy matching. 3 System Overview Our hallucination detection approach integrates task-specific prompt engineering, an LLM ensemble verification mechanism, and post-processing refinements. The system is composed of three key components: fine-tuned prompt construction, hallucination span verification through LLM ensembles, and post-processing with fuzzy matching. An overview of the full pipeline is illustrated in Figure 1 . Figure 1: Overview of our hallucination detection pipeline. 3.1 Prompt Engineering for Weak Label Generation We analyzed the validation dataset to extract annotator instructions and identify patterns, enabling the construction of a fine-grained prompt with few-shot examples. Iterative refinement improved extraction accuracy. Detailed prompt in Appendix A . 3.2 Selection of State-of-the-Art LLMs Building on the insights from the Vectara LLM Report, we chose Gemini-2.0-Flash-Exp, Qwen-2.5-Max Yang et al. 2024 , GPT-4o OpenAI 2024 , and DeepSeek-V3 Liang and et al. 2024 as our primary models for hallucination detection. These models were selected for their strong factual accuracy and reliable generation capabilities, ensuring consistent performance across multiple languages. Figure 2 illustrates the model rankings from the report. Figure 2: Performance rankings of LLMs according to the Vectara Hallucination Leaderboard Vectara 2024 . 3.3 LLM Ensemble Verification Mechanism Our hallucination detection pipeline utilizes a multi-stage ensemble verification process. With four selected LLMs—Gemini-2.0-Flash-Exp, Qwen-2.5-Max, GPT-4o, and DeepSeek-V3—we systematically rotate through different configurations, where one model identifies hallucinated spans while the other three act as adjudicators. This setup is inspired by the Mu-SHROOM annotation process, where multiple human annotators reviewed and adjudicated hallucination spans in the validation and test datasets. By simulating this human adjudication process with LLMs, we aim to improve label consistency and mitigate annotation biases. Span Extractor Model (SEM) A primary LLM identifies hallucinated spans by analyzing question-answer pairs. Given a question Q and an answer A , the span extractor outputs candidate hallucination spans S=\{s_{1},s_{2},\dots,s_{k}\} : S=\text{LLM}_{\text{extract}}(Q,A,\text{prompt}) Voting Adjudicator Models (VAMs) The three remaining LLMs act as adjudicators, independently assessing each span s_{i}\in S and assigning a hallucination probability score: p_{ij}=M_{j}(s_{i},Q),\quad p_{ij}\in[0,1] where M_{j} represents an adjudicator LLM. Iterative Model Rotation: This process is repeated for all possible combinations of the four models, ensuring that each model serves as the span extractor exactly once, while the other three act as adjudicators. Given four models, this results in a total of four unique verification runs. Consensus-Based Labeling (CBL): The final hallucination probability for each span is determined by aggregating the probabilities across all verification runs: p_{i}=\frac{1}{N}\sum_{j=1}^{N}p_{ij} where N=3 is the number of adjudicator models per run. The final hallucination label is assigned using a majority voting scheme across all runs. A span is classified as hallucinated if: p_{i}\geq 0.7 The threshold \tau=0.7 was chosen based on empirical observations on the validation set. During tuning, we found that lower thresholds (e.g., 0.5) tended to produce too many false positives by labeling uncertain spans as hallucinations, while higher thresholds (e.g., 0.8) missed subtle hallucinations annotated by human reviewers. A threshold of 0.7 offered the best trade-off between precision and recall, and its behavior closely matched the annotation patterns observed in the Mu-SHROOM validation data Vázquez et al. 2025 . This iterative model selection ensures robustness by reducing individual model biases and leveraging diverse perspectives from different LLMs. 3.4 Post-Processing with Fuzzy Matching LLMs frequently introduce minor inconsistencies in span extraction, such as variations in capitalization, extra spaces, or incomplete word boundaries. To minimize these errors, we use fuzzy matching with a similarity threshold of 0.9 (partial ratio). The similarity score between a predicted span s_{i} and a ground truth span g_{j} is

Read the complete text