Empirical Analysis and Detection of Hallucinations in LLM-Generated Bug Report Summaries

1 st Hinduja Nirujan, 2 nd Shreyas Patil, 3 rd Abdallah Ayoub, 4 th Ahmad Abdel Latif, 5 th Gouri Ginde · 2026

Read Empirical Analysis and Detection of Hallucinations in LLM-Generated Bug Report Summaries by 1 st Hinduja Nirujan, 2 nd Shreyas Patil, 3 rd Abdallah Ayoub, 4 th Ahmad Abdel Latif, 5 th Gouri Ginde (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24137.txt.

Introduction

Nirujan and four co-authors study false claims in machine-written bug-report summaries, grading each part on its own. A first look at 80 structured summaries found nearly half missing facts and about an eighth inventing facts. From Mozilla bug reports, they plant controlled fake details to build a test set, then train a detector that flags tainted summaries and names the hit parts. The best model scores well on all three jobs, and common fault patterns show where summary tools still mislead developers.

Catalog record

CREATOR1 st Hinduja Nirujan, 2 nd Shreyas Patil, 3 rd Abdallah Ayoub, 4 th Ahmad Abdel Latif, 5 th Gouri Ginde
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57005
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24137

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Empirical Analysis and Detection of Hallucinations in LLM-Generated Bug Report Summaries 1 st Hinduja Nirujan, 2 nd Shreyas Patil, 3 rd Abdallah Ayoub, 4 th Ahmad Abdel Latif, 5 th Gouri Ginde Abstract Background: Large Language Models (LLMs) are increasingly used to generate summaries of software bug reports, including sections such as Steps-to-Reproduce (S2R), Actual Behavior (AB), and Expected Behavior (EB). However, these models frequently produce hallucinations that can be convincing but are unsupported by the source bug report. This can mislead developers and reduce trust in automated software maintenance tools. Existing hallucination detection approaches typically evaluate outputs at the full-response level and do not consider the structural properties of technical documents. An initial exploratory study on 80 structured bug report summaries revealed that approximately 47.9% contained missing information, while 12.3% included fabricated content, highlighting the need for systematic hallucination analysis in bug report summarization. Aims: In this work, we empirically investigate hallucinations in LLM-generated bug report summaries from a section-aware perspective. Method: Using the BugsRepo dataset (multifaceted dataset derived from Mozilla OSS projects), we introduce controlled synthetic hallucination injection to construct a benchmark for training and evaluation. We propose a section-aware hallucination detection approach that jointly predicts whether a summary contains hallucinated content, identifies the affected bug report sections, and classifies hallucination types. Results: Experimental results across multiple pretrained language models show that the proposed approach achieves strong performance across all tasks, with the best model obtaining 0.89 report-level Macro-F1, 0.83 section-level Macro-F1, and 0.84 hallucination-type Macro-F1. We further analyze common hallucination patterns and model failure modes to better understand the limitations of current LLM-generated bug report summaries. Conclusions: The findings highlight the importance of section-aware hallucination analysis for improving the reliability of LLM-assisted bug report summarization in software maintenance workflows. I Introduction Software maintenance is a major expense in the software development process, costing billions of dollars every year [ 14 ] . For example, the Consortium for Information & Software Quality estimated that poor software quality cost the U.S. at least $2.41 trillion in 2022, with accumulated technical debt [ 16 ] . The major hurdle in the software development industry is finding and fixing software issues, including bugs, which can affect functionality and security vulnerabilities that can put the system at risk. This effort is substantial as prior work reports that developers spend, on average, about 50% of their time finding and fixing bugs [ 10 ] . The core of bug resolution relies on the bug reports, but their effectiveness relies on the details present in the bug report. Well-structured reports are clearly able to articulate actual behavior(AB), expected behavior(EB), and steps to reproduce (S2R). This structuring minimizes ambiguity and enables developers to resolve issues without much discussion and clarification [ 35 ] . With recent advancements in Natural Language Processing (NLP) and and Large Language Models (LLMs) showed strong capabilities of in various Software Engineering (SE) tasks such as bug report generation and summarization [ 3 ] . The models have demonstrated significant potential in assisting software development and maintenance activities. However, despite their promising performance, LLMs are prone to generating plausible-sounding yet factually incorrect information, commonly referred to as hallucinations . This phenomenon poses a significant challenge to the reliability and trustworthiness of LLM outputs, particularly in critical SE tasks where accuracy and faithfulness to the source information are essential. Therefore, detecting and mitigating hallucinations in LLM-generated output (e.g., bug reports) is crucial for ensuring their practical adoption in real-world SE workflows. Existing hallucination detection methods can be broadly classified into: self-consistency [ 11 ] , knowledge-graph two-stage detection [ 15 ] , fine-tuned classifiers, and LLM-as-a-Judge [ 9 ] . While these techniques have advanced the field, there has been little research specifically examining hallucinations in the SE domain. Moreover, these methods treat hallucinations as a whole and monolithically, considering them as a single class of errors. This perspective fails to capture the diversity of factual errors and overlooks how hallucination patterns can vary across different semantic sections of a bug report. As a result, these methods do not account for the distinct roles and contextual behaviors of sections. Understanding hallucination patterns is crucial, as different types of hallucinations may require different detection and mitigation strategies depending on the semantic context and functional role of each section in the bug report. Motivation. To better understand the behavior of hallucinations in LLM-generated bug report summaries, we conducted an exploratory analysis on a subset of the BugsRepo dataset [ 2 ] , a multifaceted curated dataset derived from Mozilla projects of bug reports organized into semantic fields such as S2R, AB, and EB. Specifically, we randomly selected 80 bug reports from the BugsRepo unstructured dataset, based on their structural characteristics and length. Next, we prompted Llama 3.2 using a prompt that contains explicit definitions and rules as shown in Fig. 1 , with the temperature set to 0.0 to ensure deterministic outputs. The summaries were generated under two compression settings according to the report length distribution, where low compression allowed up to 180 tokens and high compression allowed 90 tokens. Each summary was then independently and anonymously annotated by three authors using three labels: supported, incomplete, and fabricated. The agreement between the annotators was measured using Cohen’s Kappa [ 4 ] , giving a high agreement (Kappa = 0.85). After resolving disagreements through discussion among the annotators, the final labels were established. You are a software engineer summarizing a software bug report into three structured sections. Definitions • Steps to Reproduce : explicit user actions required to recreate the issue. • Actual Behavior : what actually happens when the steps are performed. • Expected Behavior : what should happen instead. Output Format (must follow exactly) Steps to Reproduce: Actual Behavior: Expected Behavior: Rules 1. Use the exact headers shown above , including the colon. 2. Use bullet points starting with “- ”. 3. Do not add explanations outside the template. 4. Do not invent information not explicitly stated in the report. 5. If a section is not explicitly stated, write exactly: Not specified. 6. Include all three sections, even if “Not specified.” Bug Report: [BUG REPORT TEXT] Fig. 1: Prompt template used to generate structured bug report summaries. Figure 2 presents the results of this analysis. From the figure, we observe that hallucinations are prevalent in LLM-generated bug report summaries, with over 60% of summary segments containing either incomplete or fabricated information. Further, hallucination rates vary across semantic sections of bug reports, suggesting that hallucination behavior is not uniform and may depend on the contextual role of each section. Fig. 2: Results of the exploratory analysis of hallucinations in LLM-generated bug report summaries. To address this gap, we propose a section-aware hallucination detection approach for structured bug report summaries. Using BugsRepo [ 2 ] subset of structured reports as the source corpus, we constructed a synthetic hallucination dataset by injecting controlled inconsistencies into specific semantic sections, then fine-tuned language models to detect hallucinated sections and classify hallucination types. More specifically, our study answers the following three research questions (RQs) that progressively address the key components required for this approach: • RQ1: How does hallucination sensitivity vary across different sections of structured bug reports during

Read the complete text