VERI-DPO: Evidence-Aware Alignment for Clinical Summarization via Claim Verification and Direct Preference Optimization
Weixin Liu, BS · 2026
Read VERI-DPO: Evidence-Aware Alignment for Clinical Summarization via Claim Verification and Direct Preference Optimization by Weixin Liu, BS (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10494.txt.
Introduction
Weixin Liu studies how to keep AI hospital summaries faithful to patient records when the claim checker itself is noisy. A verifier can reward summaries that simply say less, so the proposed VERI-DPO framework controls for coverage. It samples several summaries, splits them into claims, checks each against the evidence, and forms pairs only when one summary is better supported at similar content. Standard preference training then distills those pairs. On MIMIC test data, unsupported claims fell sharply, blinded judges preferred the output, and gains transferred to new patients.
Catalog record
| CREATOR | Weixin Liu, BS |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45113 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10494
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VERI-DPO: Evidence-Aware Alignment for Clinical Summarization via Claim Verification and Direct Preference Optimization Weixin Liu, BS Abstract Abstract Brief Hospital Course (BHC) narratives must be clinically useful yet faithful to fragmented EHR evidence 1 , 14 . LLM-based clinical summarizers still introduce unsupported statements 17 , and alignment can encourage omissions (“say-less” degeneration) 22 . We introduce VERI-DPO , which uses claim verification to mine preferences and distill them into the summarizer with DPO 20 . On MIMIC-III-Ext-VeriFact-BHC 4 , 13 (100 ICU patients; patient-level splits), we train a retrieval-augmented verifier to label claim–evidence pairs as Supported, Not Supported, or Not Addressed via a single-token format. The verifier scores sentence-level claims from sampled BHC candidates and aggregates margins into a coverage-aware utility to mine length-controlled, contradiction-anchored preference pairs. On held-out patients, verifier-mined preferences separate candidates by contradiction density, and VERI-DPO reduces Not Supported claim rates from 10.7% to 1.9% (local verifier judge) and from 11.6% to 6.4% (GPT-4o judge) 12 , while improving validity from 76.7% to 82.5% and maintaining informative length. Introduction The Brief Hospital Course (BHC) is a core narrative component of the discharge summary, intended to provide a concise yet clinically actionable account of the inpatient trajectory for handoff across care settings. 9 , 1 As such, it must be both informative enough to support downstream decision-making and faithful to what is documented in the electronic health record (EHR). 9 , 1 Automating BHC generation is difficult because the evidence needed to justify summary statements is fragmented across longitudinal notes with heterogeneous structure, variable quality, and time-dependent context. 19 A summarization model must correctly attribute diagnoses, medications, procedures, and temporal relations to the record, and avoid producing plausible but unsubstantiated statements. Even a small number of unsupported claims can have outsized impact because clinical narratives are treated as authoritative documentation and errors may propagate into follow-up care and documentation workflows. 14 Large language models (LLMs) have advanced clinical summarization via instruction tuning 18 and retrieval-augmented generation 2 , while evaluation has increasingly shifted away from surface-level similarity toward factuality-oriented assessment, motivated by evidence that overlap metrics are poorly aligned with clinically meaningful errors. 17 , 15 Nevertheless, reliably faithful long-form summaries remain challenging. Two practical barriers limit progress. First, factuality supervision is expensive and granular: verifying a BHC requires decomposing it into many statements and mapping each statement to supporting evidence, which is difficult to scale with clinician annotation. 4 Second, alignment under imperfect or indirect signals can induce shortcut behavior, such as producing shorter or vaguer narratives that make fewer checkable claims (degeneration-by-omission). 22 These issues motivate methods that operationalize statement-level verification at scale while explicitly preserving informativeness. Claim verification offers an evidence-linked framework for modeling the relationship between a claim and supporting evidence (e.g., entailment/contradiction/unknown), but in clinical text it must contend with incomplete documentation, temporal dependence, and retrieval artifacts. 23 , 24 Preference-based alignment methods, including Direct Preference Optimization (DPO) 20 , can shape generation behavior from pairwise preferences rather than token-level labels, but require preference signals that are tightly coupled to evidence to avoid omission-style shortcuts. 22 In this paper, we introduce VERI-DPO ( Verifier-Driven Direct Preference Optimization ), an evidence-aware alignment pipeline that uses a compact verifier as an auditable intermediate signal and distills verifier-driven preferences into the summarizer via DPO. 20 As illustrated in Figure 1 , the workflow has three stages: (i) fine-tune a lightweight retrieval-augmented verifier to predict Supported , Not Supported , or Not Addressed for a claim paired with patient-specific evidence using a constrained single-token decision format; (ii) sample multiple candidate BHCs for an evidence-window prompt, score sentence-level claims with the verifier, and aggregate claim-level outputs into a coverage- and length-aware utility that penalizes contradiction signals; and (iii) mine length-controlled, contradiction-anchored preference pairs and apply DPO to shift the summarizer toward evidence-consistent outputs, amortizing verifier guidance into a single-sample policy. Figure 1 : Overview of VERI-DPO . (A) Verifier training: a compact retrieval-augmented verifier predicts single-token labels A/B/C for each (evidence, claim) pair: A =Supported, B =Not Supported (error/hallucination), and C =Not Addressed (omission / not in retrieved evidence). (B) Verifier-driven preference mining: for each sampled BHC candidate, we decompose it into sentence-level claims, retrieve per-claim evidence, and score claims with the verifier; we aggregate claim-level signals into a utility U(y) that penalizes contradictions (NS/HCNS) while discouraging degeneration-by-omission, and select (y^{+},y^{-}) under utility and length/coverage constraints (HCNS: B under \mathrm{bias}_{\mathrm{prec}} with margin \Delta>\delta ). (C) DPO alignment: mined pairs are used to optimize a single-sample policy that internalizes verifier preferences, avoiding inference-time reranking. Compared with existing approaches that use factuality primarily for evaluation 17 , 15 , our approach elevates evidence-linked verification into a scalable and inspectable training signal. The verifier produces intermediate artifacts (claim-level labels, evidence identifiers, and confidence margins) that support auditing and provide a simple calibration knob for controlling the precision–recall trade-off used in preference construction. On VeriFact-BHC (100 patients; patient-level 72/8/20 train/dev/test splits), DPO distills verifier-mined preferences into a summarizer that reduces unsupported-claim rates under both a local verifier judge and an external GPT-4o claim-level judge while maintaining validity and informative length. 12 We summarize our major contributions as follows: • We develop a retrieval-augmented, lightweight clinical claim verifier trained with patient-level splits, and evaluate accuracy and calibration on held-out patients. • We introduce verifier-driven preference mining for long-form clinical summaries, including high-confidence contradiction anchoring and constraints that control length and claim-coverage mismatch. • We apply Direct Preference Optimization (DPO) on mined pairs to distill verifier supervision into a single-sample summarizer, achieving reduced unsupported-claim rates under dual-judge evaluation while mitigating degeneration-by-omission. 22 Methods Data and Preprocessing Dataset. We use the MIMIC-III-Ext-VeriFact-BHC dataset 4 under credentialed PhysioNet access 6 for patient-specific factual verification of long-form clinical summaries against each patient’s EHRs. The dataset is a de-identified derivative of the MIMIC-III Clinical Database v1.4. 13 It contains 100 patients (125 hospital admissions) and two BHC narratives per patient: (i) a human-written BHC extracted from the discharge summary and (ii) an LLM-generated BHC produced by summarizing the patient’s longitudinal notes. 4 The patient-specific reference EHRs exclude the final discharge summary note by dataset construction and include 4,787 clinical notes in total (median 25 notes per patient; IQR 16–47; range 10–314). Note categories span 14 types, with the most frequent being Nursing (1,228), Physician (1,223), Radiology (1,134), Respiratory (471), and electrocardiogram (ECG) (306). Proposition-level supervision. Each BHC narrative is decomposed into proposition-level units at two granularities: sentence propositions and atomic claim propositions (single predicate-like statements), yielding 13,070 proposition instances. 4 Table 1 provides paraphrased examples illustrating the proposition format and three-way labels. Reference EHRs and labels. Clinicians label extracted propositions as Supported , Not Supported , or Not Addressed with respect to each patient’s reference EHRs; ground truth is established via majority vote across three clinicians with adjudication for disagreements. 4 In verifier preprocessing, we treat each labeled proposition as a candidate training instance (retaining proposition_type for analysis), apply claim-validity filtering before retrieval/JSONL construction, and exclude note row identifiers associated with discharge-summary sources used to derive propositions. In this dataset release, the reference EHRs already exclude the final discharge summary note; we verify that the additional row-id exclusion does not remove extra reference notes. Table 1 :