Hallucination Detection-Guided Preference Optimization for Clinical Summarization

Shamanth Kuthpadi Seethakantha, Dung Ngoc Thai, Vara Prasad Gudi, Simran Tiwari, Rami Matar, Avijit Mitra, Wenlong Zhao, Andrew McCallum, Wael Salloum, [0.5em] Manning College of Information, Computer Sciences · 2026

Read Hallucination Detection-Guided Preference Optimization for Clinical Summarization by Shamanth Kuthpadi Seethakantha, Dung Ngoc Thai, Vara Prasad Gudi, Simran Tiwari, Rami Matar, Avijit Mitra, Wenlong Zhao, Andrew McCallum, Wael Salloum, [0.5em] Manning College of Information, Computer Sciences (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28910.txt.

Introduction

Shamanth Seethakantha and colleagues fight made-up statements in clinical summaries from large language models. Their HDSR method revises drafts step by step, steered by detectors that flag claims with no support. A second method, HDSR-PL, turns those revision paths into preference pairs for tuning the model itself. Tests on real MIMIC-IV notes show steep drops in made-up statements for Llama and Gemma models, while human and automated judges rate fluency, coherence, and relevance as preserved.

Catalog record

CREATORShamanth Kuthpadi Seethakantha, Dung Ngoc Thai, Vara Prasad Gudi, Simran Tiwari, Rami Matar, Avijit Mitra, Wenlong Zhao, Andrew McCallum, Wael Salloum, [0.5em] Manning College of Information, Computer Sciences
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51086
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28910

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hallucination Detection-Guided Preference Optimization for Clinical Summarization Shamanth Kuthpadi Seethakantha, Dung Ngoc Thai, Vara Prasad Gudi, Simran Tiwari, Rami Matar, Avijit Mitra, Wenlong Zhao, Andrew McCallum, Wael Salloum, [0.5em] Manning College of Information, Computer Sciences Abstract Large language models (LLMs) have shown promise on summarization tasks, but they often produce hallucinations, which are unsupported or incorrect statements that limit their reliability in specialized healthcare applications. We introduce Hallucination Detection guided Self-Refinement ( HDSR ), an inference-time method that leverages hallucination detectors to guide iterative summary revisions toward factual corrections. Building on this, we propose HDSR for Preference Learning ( HDSR-PL ), which converts detector-guided refinement trajectories into preference pairs for model finetuning. Extensive experiments show that our methods substantially reduce hallucinations for Llama and Gemma models in summarizing real-world clinical notes from MIMIC-IV-Note v2.2 . For example, HDSR reduces 24% and HDSR-PL reduces 48% hallucinations in Llama-3.1-8B-Instruct. Importantly, both methods preserve summary fluency, coherence, and relevance according to human expert and LLM-Jury evaluations. Together, these results demonstrate that detection-informed refinement and preference learning offer an automated solution for improving factual faithfulness in clinical summarization. 1 Introduction Figure 1: Overview of hallucination mitigation via detection-informed self-refinement. Given an input clinical note, a language model generates an initial summary that may contain unsupported or hallucinated medical content. A hallucination detector identifies unsupported content, which is used to guide iterative self-refinement toward removing factual errors rather than stylistic changes ( top; HDSR ). The intermediate outputs from HDSR process are converted into preference pairs and used for preference learning (e.g., DPO), amortizing faithful behavior and yielding hallucination-mitigated summaries at inference time ( bottom; HDSR-PL ). Large language models achieve strong summarization performance but often generate hallucinations, defined as content not supported by the source or inconsistent with world knowledge ( 16 ; 9 ; 21 ) . Hallucinations remain prevalent even in recent LLMs, with rates of roughly 40-50% on benchmarks such as FaithBench ( 2 ) indicating persistent factual unreliability. This issue is critical in clinical summarization, where LLMs condense long patient records to support care delivery. Despite comparable performance with clinician - written summaries ( 22 ) , hallucinations manifest as fabricated or distorted clinical statements that are hard to detect ( 1 ; 11 ) . Even domain - adapted models frequently introduce hallucinations expressed with medically plausible terminology, requiring expert review ( 8 ; 24 ; 5 ; 3 ) , and underscoring the need for stricter factual reliability in clinical summarization than in general - domain tasks. Existing hallucination mitigation strategies are broadly divided into training-time and inference-time approaches. Training-time methods include domain pretraining, continual pretraining, supervised fine-tuning, parameter-efficient adapters ( 22 ; 25 ) , and specialized loss functions or reinforcement learning objectives ( 4 ; 2 ; 1 ) . These methods require large volumes of high-quality domain data ( 8 ) and depend on factuality metrics that correlate weakly with clinical correctness, limiting their practical impact. Reinforcement learning from human or AI feedback offers greater flexibility, but requires carefully engineered preference data that is difficult to scale in medical settings ( 14 ) . Inference - time techniques avoid altering model parameters: retrieval - augmented generation grounds outputs in external documents yet depends on retrieval quality ( 12 ; 23 ; 11 ) , while self - refinement ( 15 ) and verification loops ( 21 ) iteratively critique and revise generations, improving factuality at the cost of higher inference overhead and a risk of over - editing. Recent work combining iterative refinement with alignment shows such approaches can strip salient content or introduce new hallucinations even when fluency is preserved ( 10 ) . A closely related approach is SynFac-Edit ( 19 ) , which generates synthetic edit feedback for preference optimization but relies on predefined error types and external edit models. Our pipeline combines hallucination detection with iterative self refinement, using detector feedback to guide self refinement and to form preference pairs from the original and revised summaries. We then train LLMs using direct preference optimization on these preference pairs, producing models that generate more faithful summaries with fewer hallucinations. On the MIMIC IV clinical summarization hallucination datasets ( 8 ) , our method reduces hallucinations by approximately 24% through self refinement and by up to 48% after fine tuning, while preserving summary quality and fluency according to both human and LLM-as-judge evaluations, and without incurring additional inference time cost. 2 Method Figure 1 illustrates our pipeline, which uses hallucination detection feedback to guide summary revision and train models to prefer faithful summaries. 2.1 Problem Statement Given a source document and a generated summary, the goal of hallucination mitigation is to produce summaries that are informative while remaining faithful to the source. We view a source document and its summary as describing clinical facts documented during a patient’s hospital course. A summary is faithful if these facts are supported by the source document. Hallucinations arise when a summary introduces unsupported facts or adds procedures or findings that were not documented, for example when “leg ultrasound negative” appears in the summary without support from the source. 2.2 Hallucination Detection Guided Self-Refinement During self-refinement, a hallucination detector is applied to the generated summary to identify content that is unsupported or inconsistent with the source document. Detector feedback highlights unsupported spans or statements, and the model is prompted to revise these parts while preserving supported content (see Appendix A.2 ). This process focuses revisions on factual corrections rather than stylistic changes. Detection and revision are applied iteratively. After each revision, the updated summary is re-evaluated by the detector, and the resulting feedback guides subsequent revisions until a fixed iteration limit is reached or no further hallucinations are detected. This detector guided refinement improves factual alignment while maintaining overall fluency. We use existing hallucination detectors from prior work, specifically MedCat ( 13 ) and prompt based detectors following the MedAlign annotation guidelines ( 6 ) . MedCat links clinical concepts in the source document and the summary to biomedical ontologies in order to flag unsupported or missing content. MedAlign defines a clinically motivated taxonomy of hallucination categories, such as unsupported procedures or medications, which we implement as a prompted detector. 2.3 Preference Learning from Detection Guided Self-Refinement Detection-guided self-refinement produces pairs of summaries for each source document, consisting of an initial summary and a revised summary with fewer hallucinations. We treat the revised summary as the preferred output and the initial summary as the non-preferred output, thereby forming preference pairs without human annotation. We train the summarization model using direct preference optimization on these pairs. This objective encourages the model to internalize detector guided factual corrections, amortizing improvements from self refinement into the model parameters and enabling faithful generation at inference time without additional refinement steps. 3 Experiments 3.1 Experimental Setup We study clinical summarization from Brief Hospital Course (BHC) sections to Discharge Instructions (DI) using datasets derived from MIMIC-IV-Note v2.2 ( 8 ) . Experiments are evaluated on the hallucination-annotated subset Hallucination-Generated-DI , following the task formulation and annotation guidelines of 8 . We include GPT-5 ( 20 ) zero-shot as a closed-source reference and primarily focus on LLaMA-3.1-8B-Instruct ( 18 ) as the open-source base

Read the complete text