RadFlag: A Black-Box Hallucination Detection Method for Medical Vision Language Models

Serena Zhang, Sraavya Sambara, Oishi Banerjee, Julian N. Acosta, L. John Fahrner, Pranav Rajpurkar · 2024

Read RadFlag: A Black-Box Hallucination Detection Method for Medical Vision Language Models by Serena Zhang, Sraavya Sambara, Oishi Banerjee, Julian N. Acosta, L. John Fahrner, Pranav Rajpurkar (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00299.txt.

Description

Abstract Generating accurate radiology reports from medical images is a clinically important but challenging task. While current vision language models show promise, they are prone to generating hallucinations, potentially compromising patient care. We introduce RadFlag, a black-box method to enhance the accuracy of radiology report generation.

Catalog record

CREATORSerena Zhang, Sraavya Sambara, Oishi Banerjee, Julian N. Acosta, L. John Fahrner, Pranav Rajpurkar
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RadFlag: A Black-Box Hallucination Detection Method for Medical Vision Language Models Serena Zhang, Sraavya Sambara, Oishi Banerjee, Julian N. Acosta, L. John Fahrner, Pranav Rajpurkar Abstract Generating accurate radiology reports from medical images is a clinically important but challenging task. While current vision language models show promise, they are prone to generating hallucinations, potentially compromising patient care. We introduce RadFlag, a black-box method to enhance the accuracy of radiology report generation. Our method uses a sampling-based flagging technique to find hallucinatory generations that should be removed. We first sample multiple reports at varying temperatures and then use a large language model to identify claims that are not consistently supported across samples, indicating that the model has low confidence in those claims. Using a calibrated threshold, we flag a fraction of these claims as likely hallucinations, which should undergo extra review or be automatically rejected. Our method achieves high precision when identifying both individual hallucinatory sentences and reports that contain hallucinations. As an easy-to-use, black-box system that only requires access to a model’s temperature parameter, RadFlag is compatible with a wide range of radiology report generation models and has the potential to broadly improve the quality of automated radiology reporting. These authors contributed equally. † † volume: 259 † † year: 2024 † † submitted: LEAVE UNSET † † published: LEAVE UNSET † † workshop: Machine Learning for Health (ML4H) 2024 keywords Hallucination Detection, Radiology Report Generation, Selective Prediction, Chest X-Rays, Large Language Models, Vision Language Models, Uncertainty Estimation Data and Code Availability The primary dataset used in this research is MIMIC-CXR, an open-source chest X-ray report generation dataset publicly available through PhysioNet ( Johnson et al., 2019 ) . Code will be made publicly available at https://github.com/rajpurkarlab/RadFlag . 1 Introduction Generating accurate radiology reports from medical images is a challenging task that requires a deep understanding of medical imaging and the ability to precisely interpret and communicate complex findings. Medical Vision Language Models (VLMs) that automatically generate radiology reports from chest X-ray images have the potential to mitigate shortages of radiological expertise and improve clinical efficiency and accuracy ( Bannur et al., 2024 ; Tanida et al., 2023 ; Chen et al., 2024 ) . However, such models are prone to hallucinations, where the system generates incorrect claims that are not supported by the input image ( Liu et al., 2024a ; Bai et al., 2024 ) . For example, even high-performing radiology report generation models can hallucinate in approximately 40% of generated sentences (see Table 2 in Appendix D ). Such errors can mislead clinicians, with potentially severe consequences for patient care. To reduce the adverse impact of hallucinations, we propose RadFlag, a new, easy-to-use method for detecting hallucinations in AI-generated radiology reports without needing access to the AI model’s inner workings. RadFlag is the first black-box hallucination detection method specifically designed for VLMs, and flags both sentences and reports as containing hallucinated findings. We provide two key contributions: Figure 1: RadFlag flags hallucinatory text in a candidate report by comparing it against a corpus of additional reports sampled at a higher temperature. GPT-4 is used to count how many of the high-temperature reports support each sentence in the candidate report. We flag sentences with little support, as well as reports that contain multiple flagged sentences. • We develop a novel entailment scoring method specifically designed for radiology, which accurately identifies medical findings that a VLM has low confidence in. We combine this scoring method with conformal risk control thresholding to detect hallucinated sentences with high precision. • We further aggregate the sentence-level flagging results to identify reports with a high rate of hallucinations. This analysis enables selective prediction at the report level, making it possible to abstain from generating reports when there is a high chance of hallucinating. Our empirical results show that RadFlag can accurately flag 28% of hallucinatory sentences while maintaining a flagging precision of 73% on Medversa, a recent high-performing report generation model Zhou et al. (2024) . At the report level, our method analyzed 208 reports generated by MedVersa and divided them into two sets: a flagged set with 4.2 hallucinations per report (n = 57) and an accepted set with only 1.9 hallucinations per report (n = 151). This approach can unlock selective prediction, where models refrain from generating reports in cases of heightened uncertainty. 2 Related Work 2.1 Uncertainty Modeling and Hallucination Mitigation Techniques for uncertainty modeling in LLMs fall into three categories based on how much access to the model is required: white-box (using internal components such as intermediate activations), gray-box (using output probabilities), and black-box (using only generated outputs) ( Liu et al., 2024b ; Huang et al., 2023 ) . Recently, black-box methods measuring self-consistency have emerged as accurate, easy-to-use approaches to assess model output confidence ( Zhang et al., 2024 ; Mündler et al., 2024 ) . Self-Check GPT, a black-box hallucination detection method, compares multiple sampled responses, measures their consistency using tools like BERTscore and LLM prompting, and labels claims that appear inconsistently across samples as hallucinations. Despite using only model outputs, this black-box method demonstrates superior performance compared to gray-box approaches ( Manakul et al., 2023 ) . Discrete semantic entropy offers another black-box approach that clusters multiple generated outputs based on semantic similarity and computes entropy over these clusters to measure uncertainty ( Farquhar et al., 2024 ) . While these methods have proven effective for text-only models, to the best of our knowledge, RadFlag is the first work to extend this concept to VLMs. In the domain of radiology report generation, various techniques have been explored to mitigate hallucinations and improve accuracy. Existing methods are generally not black-box and often involve fine-tuning a model with new losses or datasets ( Banerjee et al., 2024 ; Anonymous, 2024 ; Ramesh et al., 2022 ) . Additionally, the intersection of uncertainty modeling and hallucination detection remains relatively unexplored for radiology report generation. To the best of our knowledge, only one study has applied uncertainty modeling to remove hallucinations in radiology report generation by using Monte Carlo dropout to estimate uncertainty at both the image and text levels, integrating these estimates into a novel loss function ( Wang et al., 2024 ) . While effective, this method relies on access to model weights. In contrast, our sampling-based method provides a versatile, easy-to-use solution that is model-agnostic and compatible with both proprietary and closed-source models. 2.2 Selective Prediction for Abstention The concept of selective prediction, where models abstain from generating predictions when there is a high chance of error, has also been explored in Natural Language Processing (NLP) ( Geifman and El-Yaniv, 2017 ; Xin et al., 2021 ; Vazhentsev et al., 2023 ) . Selective prediction relies on uncertainty estimation to decide whether to withhold a response, using the assumption that the model is likely to be incorrect when uncertain. A study by Yadkori et al. (2024) reduces hallucinations in LLM question-answering by combining selective prediction with self-consistency sampling. They evaluate response similarity across samples and use a calibration dataset to set a selective prediction threshold, offering strong theoretical guarantees on error rates. Selective prediction has also been applied to medical imaging. In chest X-ray image classification,

Read the complete text