Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation?
Evangelia Gogoulou Shorouq Zahra Liane Guillou Luise Dürlich Joakim Nivre · 2025
Read Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation? by Evangelia Gogoulou Shorouq Zahra Liane Guillou Luise Dürlich Joakim Nivre (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20699.txt.
Introduction
Evangelia Gogoulou and co-authors ask a 2025 question about language models: can they spot made up content in their own kind of output? Models often produce text that is false or makes no sense, a flaw called hallucination. The team tested open models on finding such flaws in two tasks, translation and paraphrasing. The flaws at issue conflict with the source text, rather than with the outside world.
Performance varied across models, but stayed steady across different prompts. The team also studied the effects of model size and of instruction tuning. Simpler models built for logic checks did about as well as the large models. That result suggests large model detectors are not the only workable option for this task. The study builds on a recent shared task for hallucination detection.
Catalog record
| CREATOR | Evangelia Gogoulou Shorouq Zahra Liane Guillou Luise Dürlich Joakim Nivre |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20699
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation? Evangelia Gogoulou Shorouq Zahra Liane Guillou Luise Dürlich Joakim Nivre Abstract A frequently observed problem with LLMs is their tendency to generate output that is nonsensical, illogical, or factually incorrect, often referred to broadly as “hallucination”. Building on the recently proposed HalluciGen task for hallucination detection and generation, we evaluate a suite of open-access LLMs on their ability to detect intrinsic hallucinations in two conditional generation tasks: translation and paraphrasing. We study how model performance varies across tasks and language and we investigate the impact of model size, instruction tuning, and prompt choice. We find that performance varies across models but is consistent across prompts. Finally, we find that NLI models perform comparably well, suggesting that LLM-based detectors are not the only viable option for this specific task. *[inlinelistone,1]label=( 0) , *[inlinelisttwo,1]label=( ) , 1 Introduction The introduction of large language models (LLMs) has revolutionised the field of natural language processing (NLP). State-of-the-art LLMs have demonstrated excellent language generation capabilities in conversational AI Zhao et al. (2024) , as well as strong performance on more specific NLP tasks like summarisation Pu et al. (2023) , open-domain question answering Kamalloo et al. (2023) , sentiment analysis Zhang et al. (2024) , and machine translation Kocmi et al. (2023) . Despite this success, LLMs are prone to producing output that is fluent and grammatical but semantically inadequate or factually incorrect, a phenomenon broadly referred to within the NLP community as “hallucination”. The impact of hallucinations by LLMs may be severe in downstream applications where accurate output is mission critical, or where hallucination leads to erroneous decisions with negative consequences that directly impact humans e.g. in the medical or legal domain. In many cases, it may be infeasible to have a human in the loop, or it may be difficult for humans to identify hallucinations, which motivates the need for automated methods for detection and evaluation. In this paper, we aim to discover whether LLMs can be used to detect hallucinated content, focusing on a special case of what Ji et al. (2023) call intrinsic hallucinations, that is, cases where the output is deficient with respect to a particular input and where the deficiency can be detected given only the input and output. 1 1 1 This in contrast to extrinsic hallucinations, where additional information such as world knowledge is required to detect the deficiency. More precisely, for the tasks of paraphrasing and machine translation, we define a hallucination to be an output, or hypothesis, that is not entailed by the input, or source. We build upon previous work from the ELOQUENT Lab at CLEF 2024 Dürlich et al. (2024) , where the HalluciGen task asked participants to apply LLMs in detecting and generating hallucinations. We extend the work from the shared task with a series of experiments in prompting open-access LLMs to detect hallucinations. It is framed as a contrastive challenge task: given a source sentence, and a pair of hypotheses, the model should detect which one contains a hallucination. We evaluate a range of open-access LLMs on hallucination detection in paraphrase generation and translation, as defined in the HalluciGen task Dürlich et al. (2024) . Through a systematic investigation of model performance on the hallucination detection task, we address the following questions: • How does model performance differ across target languages? • Does increased model parameter size improve performance? • Does instruction tuning improve performance? • Does the language and formulation of the prompt matter? 2 Background and Related Work Two concepts that are often used to characterize different types of hallucinations are faithfulness and factuality . Faithfulness means being consistent with a given source or input, and has long been used as an evaluation criterion in conditional generation tasks like machine translation; a faithfulness hallucination is therefore any output that lacks such consistency, regardless of whether it is factually correct. By contrast, factuality means corresponding to real-world knowledge, and a factuality hallucination is therefore any output that makes a false claim, regardless of context and input. A related distinction is made between intrinsic and extrinsic hallucinations, where the former can be detected by only considering the input and output of a system, while the latter requires access to additional information ( Ji et al., 2023 ) . Prior work has mostly focused on building systems to detect factuality hallucinations. For example, Li et al. (2023) introduce a benchmark targeting cases of factual hallucinations in the context of question-answering, knowledge-grounded dialogue, and summarisation. Aside from the HalluciGen task the closest work to ours is the SHROOM shared task Mickus et al. (2024) from SEMEVAL 2024. SHROOM defines hallucinations as cases when the hypothesis cannot be inferred from its semantic reference. Despite the similarity with our definition, there is a significant difference in how the hallucinations are constructed. In SHROOM they are generated by models prompted to solve the specific task scenario, whilst we mostly construct hallucinations manually based on specific categories of errors; by switching gender, negation, or tense, replacing words with their antonyms, by substituting named entities, numbers, dates, and currencies, and by making superfluous additions. The two tasks also differ in terms of their coverage of NLP tasks and target languages. SHROOM includes the additional task of definition modeling; HalluciGen covers an extra language for paraphrase but has limited coverage for machine translation. There is limited evidence so far on the effectiveness of using LLMs for detecting hallucinations. Li et al. (2023) find that LLMs, including Llama2 and ChatGPT, perform poorly on the task of identifying hallucinations that have been generated by LLMs to be factually incorrect, in English question-answering and summarisation. According to the HalluciGen task results Dürlich et al. (2024) , GPT-4 and LLM majority voting approaches outperform smaller English-centric models such as Llama3-8b and Gemma-7b. Similar conclusions emerge from SHROOM, where submissions based on GPT-4 or model ensembling exhibit the strongest performance. Model fine-tuning on SHROOM training data is another successful approach. NLI models have also shown promising abilities at detecting hallucinations. Manakul et al. (2023) propose SelfCheckGPT to detect sentence-level hallucinations using generative LLM prompting, LLM probabilities, and NLI models. Interestingly, their experimental results show that LLM prompting outperforms the NLI-based method only by a small margin, and both outperform all other SelfCheckGPT methods and baselines. Additionally, NLI-based methods yield promising results for high-resource languages in multilingual setups, often outperforming other lexical metrics (like ROUGE), especially for intrinsic hallucinations where the hypothesis would clearly contradict the source Kang et al. (2024) . The ability of NLI models to detect intrinsic hallucinations is arguably unsurprising as they must “handle phenomena like lexical entailment, quantification, coreference, tense, belief, modality, and lexical and syntactic ambiguity” ( Williams et al., 2018 ) to successfully predict entailment, contradiction, and neutral relations between sentence pairs. 3 Dataset Description The HalluciGen detection task Dürlich et al. (2024) covers the two following scenarios: • Paraphrase Generation : The model is presented with two possible paraphrases of a given source sentence in English (en) and Swedish (sv). • Machine Translation : Given a sentence in a source language, the model is presented with two possible translations in the target language; English-German (en \Leftrightarrow de)