ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports
Romain Hardy, Sung Eun Kim, Du Hyun Ro, Pranav Rajpurkar · 2024
Read ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports by Romain Hardy, Sung Eun Kim, Du Hyun Ro, Pranav Rajpurkar (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15264.txt.
Introduction
Romain Hardy, Sung Eun Kim, Du Hyun Ro, and Pranav Rajpurkar build a checker for false claims in AI-written X-ray reports. Their 2024 paper names the tool ReXTrust. False lines in a report could steer care the wrong way, so each claim needs a risk score of its own.
The tool reads the inner states of a large vision and language model as it writes. It turns those states into a risk score for each finding in the report. On a chest X-ray data set, the tool beat older methods at spotting false claims. Scores were strong for the claims that matter most for care. The authors see it as a safety check for machine-written reports.
Catalog record
| CREATOR | Romain Hardy, Sung Eun Kim, Du Hyun Ro, Pranav Rajpurkar |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15264
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports Romain Hardy, Sung Eun Kim, Du Hyun Ro, Pranav Rajpurkar Abstract The increasing adoption of AI-generated radiology reports necessitates robust methods for detecting hallucinations—false or unfounded statements that could impact patient care. We present ReXTrust, a novel framework for fine-grained hallucination detection in AI-generated radiology reports. Our approach leverages sequences of hidden states from large vision-language models to produce finding-level hallucination risk scores. We evaluate ReXTrust on a subset of the MIMIC-CXR dataset and demonstrate superior performance compared to existing approaches, achieving an AUROC of 0.8751 across all findings and 0.8963 on clinically significant findings. Our results show that white-box approaches leveraging model hidden states can provide reliable hallucination detection for medical AI systems, potentially improving the safety and reliability of automated radiology reporting. 1 Introduction The automated generation of radiology reports using large vision-language models (LVLMs) has recently shown promising results, offering the potential to improve workflow efficiency and standardization in clinical settings ( Bannur et al. 2024 ; Tanida et al. 2023 ; Hamamci, Er, and Menze 2024 ; Liu et al. 2024a ; Gu et al. 2024 ; Chen et al. 2024d ; Zhou et al. 2024 ) . However, these models sometimes generate hallucinations—statements that are false, unfounded, or inconsistent with the input images. In medical settings, such hallucinations pose significant risks, as false pathological findings could lead to unnecessary interventions or missed diagnoses, directly impacting patient care. In this paper, we introduce ReXTrust, a white-box model designed to detect hallucinations in LVLM-generated radiology reports. ReXTrust uses a self-attention module trained on LVLM hidden states, enabling fine-grained insights into specific radiological findings and reliable overall hallucination risk scores. By analyzing internal model representations, ReXTrust can identify potential hallucinations during the generation process itself, rather than relying solely on post-hoc analysis. We evaluate ReXTrust on a subset of the MIMIC-CXR dataset, demonstrating its effectiveness across different medical categories and severity levels, with particular attention to clinically significant findings that would impact patient care. Our contributions are threefold: 1. We develop a white-box architecture for radiology finding hallucination detection that enables both token-level attention analysis and finding-level prediction, making the detection process transparent to clinical users. 2. We demonstrate state-of-the-art performance in detecting hallucinations in radiology reports, particularly for clinically significant findings, achieving superior results compared to existing approaches while maintaining interpretability. 3. We provide empirical evidence that model hidden states contain reliable signals for hallucination detection, suggesting a path toward improved semantic fidelity in medical report generation systems. 2 Background and Related Work 2.1 Hallucination Detection Hallucination detection is the task of identifying AI-generated content that is false, unfounded, or inconsistent with input data. In medical report generation, hallucinations can manifest as either fictional findings or critical omissions, both of which pose significant risks to patient care. Hallucinations are particularly consequential in radiology, as false findings may both initiate unnecessary medical procedures and mask critical pathologies requiring urgent intervention. Current hallucination detection methods for LLMs and LVLMs can be categorized into three approaches based on their required level of access to model parameters: black-box, gray-box, and white-box approaches. Each category presents different tradeoffs between computational complexity, ease of implementation, and detection accuracy. Our work builds upon these foundations by introducing a white-box approach specifically designed for the radiology domain, where the stakes of hallucination detection are high. 2.2 Black-Box Methods Black-box methods operate without access to model parameters, relying solely on model outputs. Despite this limitation, these methods have gained prominence due to their broad applicability across proprietary models and APIs. Research in this area has pursued two main directions. The first explores models’ self-evaluation capabilities, with Kadavath et al. 2022 and Lin, Hilton, and Evans 2022 demonstrating that LLMs can identify their own hallucinations with reasonable accuracy when explicitly prompted. The second direction quantifies model uncertainty by analyzing output diversity. Kuhn, Gal, and Farquhar 2023 and Farquhar et al. 2024 showed that semantic entropy—the uncertainty in the meanings of model-generated text—correlates strongly with hallucination likelihood. Building on these foundations, Friel and Sanyal 2023 proposed using auxiliary LLMs as external validators, demonstrating improved detection performance across both open and closed-domain tasks. More recently, Manakul, Liusie, and Gales 2023 introduced SelfCheck GPT, which evaluates generated sequences through mutual entailment analysis across multiple generations, providing a more robust assessment of content reliability. While these methods provide valuable insights, our work demonstrates that access to model parameters can facilitate robust hallucination detection in the medical domain. 2.3 Gray-Box Methods Gray-box methods leverage access to token-level probability distributions, enabling more precise analysis than black-box approaches while remaining computationally efficient. Traditional metrics in this category include perplexity, token entropy, and mutual information ( Fomicheva et al. 2020 ; Van der Poel, Cotterell, and Meister 2022 ) . However, token-level uncertainty evaluation presents inherent challenges, as large uncertainty values may reflect the presence of multiple viable continuations rather than potential fabrications. To address this limitation, Fadeeva et al. 2024 developed a claim-conditioned scoring system (CCP) that evaluates model generations according to the probability that a semantically equivalent output would be generated instead, providing a more reliable measure of content veracity. While gray-box methods offer valuable insights into language model uncertainty, our approach moves beyond probability distributions to leverage the rich information contained in hidden states, enabling more nuanced hallucination detection capabilities. 2.4 White-Box Methods White-box methods require complete access to model weights and typically leverage intermediate representations of the model inputs. The predominant approach involves training supervised classifiers on extracted model activations. For instance, Azaria and Mitchell 2023 and Su et al. 2024 employed feedforward neural networks for post-generation hallucination detection, while Alnuhait et al. 2024 developed similar architectures for pre-generation detection and mitigation. In a significant advancement, Kossen et al. 2024 demonstrated that a logistic regression model trained on hidden activations can effectively predict semantic entropy, suggesting that LLMs encode uncertainty information within their internal representations. Building on this insight, Chen et al. 2024a introduced the EigenScore metric, which analyzes the consistency of hidden state embeddings across multiple generations to provide a robust measure of output reliability. ReXTrust extends previous white-box approaches by using a self-attention mechanism that analyzes sequences of hidden states, thus achieving high fidelity on finding-level hallucination detection. 2.5 Multimodal Methods Given that LVLMs incorporate language model components, hallucination detection methods developed for LLMs can be naturally extended to LVLMs ( Li et al. 2024b ) . However, the visual component of LVLMs presents unique opportunities to validate generated content against input images, particularly for verifying visual claims. Recognizing this potential, Yin et al. 2023 developed Woodpecker, which employs auxiliary object detectors and visual question-answering models to verify visual assertions in model-generated text. Similarly, Chen et al. 2024c approached LVLM hallucination detection through independent visual evidence-gathering modules that systematically confirm or contradict claims extracted from model generations. This approach was further refined by Fei et al. 2024 , who employed Siamese networks to compare scene graphs extracted from model generations with those derived from input images. In the medical domain, specialized LVLM hallucination detection methods have emerged to address domain-specific challenges. Chen et al. 2024b introduced MediHallDetector for categorizing medical hallucinations, demonstrating superior consistency compared to GPT baselines