Conformal Risk Control for Pulmonary Nodule Detection
Roel Hulsman, Valentin Comte, Lorenzo Bertolini, Tobias Wiesenthal, Antonio Puertas Gallardo, Mario Ceresa · 2024
Read Conformal Risk Control for Pulmonary Nodule Detection by Roel Hulsman, Valentin Comte, Lorenzo Bertolini, Tobias Wiesenthal, Antonio Puertas Gallardo, Mario Ceresa (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20167.txt.
Introduction
Roel Hulsman and co-authors add risk bounds to a lung nodule spotting model, in this 2024 paper. Lung screening seeks small nodules that may be early cancer. A model that spots them must also say how sure it is, so doctors can trust its output.
Their method gives prediction sets with formal bounds on error. Users can trade more false alarms for a firmer promise on misses. On nodules that at least three doctors had marked, the model matched the hit rate of single doctors, with a slight rise in false alarms. The paper also warns that doctors may disagree on what counts as a nodule, which clouds any fixed truth for tests.
Catalog record
| CREATOR | Roel Hulsman, Valentin Comte, Lorenzo Bertolini, Tobias Wiesenthal, Antonio Puertas Gallardo, Mario Ceresa |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-20167
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conformal Risk Control for Pulmonary Nodule Detection Roel Hulsman, Valentin Comte, Lorenzo Bertolini, Tobias Wiesenthal, Antonio Puertas Gallardo, Mario Ceresa Abstract Quantitative tools are increasingly appealing for decision support in healthcare, driven by the growing capabilities of advanced AI systems. However, understanding the predictive uncertainties surrounding a tool’s output is crucial for decision-makers to ensure reliable and transparent decisions. In this paper, we present a case study on pulmonary nodule detection for lung cancer screening, enhancing an advanced detection model with an uncertainty quantification technique called conformal risk control (CRC). We demonstrate that prediction sets with conformal guarantees are attractive measures of predictive uncertainty in the safety-critical healthcare domain, allowing end-users to achieve arbitrary validity by trading off false positives and providing formal statistical guarantees on model performance. Among ground-truth nodules annotated by at least three radiologists, our model achieves a sensitivity that is competitive with that generally achieved by individual radiologists, with a slight increase in false positives. Furthermore, we illustrate the risks of using off-the-shelve prediction models when faced with ontological uncertainty, such as when radiologists disagree on what constitutes the ground truth on pulmonary nodules. 1 1 1 Work performed while affiliated with the Joint Research Centre, before moving to Amsterdam. Code is available at https://github.com/roelhulsman/ConformalNoduleDetection . † † proceedings: PMLR: ArXiv Preprint keywords Pulmonary Nodule Detection, AI-Based Decision Support, Conformal Risk Control, Ontological Uncertainty 1 Introduction Lung cancer is the leading cause of cancer-related deaths worldwide ( Torre et al. 2016 ) . To improve survival rates, low-dose thoracic CT screening offers potential for the early detection of high-risk individuals ( Aberle et al. 2011 ) . In this context, AI-assisted detection of malignant nodules could serve as a meaningful instrument to implement CT screening on a larger scale. In fact, a recent survey among radiologists in the United States identified lesion detection as the top answer to what they would like AI to do to enhance their clinical practices ( Allen et al. 2021 ) , while also highlighting the need for methods to evaluate and report the performance of AI systems on representative datasets. Hendrix et al. 2023 presents a state-of-the-art example of such a system, showing reliable detection of benign and malignant pulmonary nodules in clinically indicated CT scans. A trustworthy representation of predictive uncertainty is crucial to evaluating the performance of AI systems, and it should be regarded a key feature in safety-critical healthcare applications ( Begoli et al. 2019 ; Chua et al. 2023 ) . In the case of AI-assisted pulmonary nodule detection, a performance assessment measures sensitivity, outlining the number of undetected nodules (that is, false negatives), as well as precision, constraining the number of false positives. A system that too often fails to detect nodules cannot be considered a safe system for large-scale clinical use, while a system that returns too many false positives has limited practical benefit for radiologists. Common practice in AI-assisted pulmonary nodule detection is to evaluate the performance of an AI system using free-response receiver operating characteristic (FROC) analysis ( Chakraborty 2013 ; Van Ginneken et al. 2010 ; Gu et al. 2021 ) , where the sensitivity of an AI system on a particular dataset is offset against the average number of false positives per scan. Each point on a FROC plot is determined by a confidence threshold \lambda , indicating the degree of confidence for a system to distinguish nodules from non-nodules. Typically, an AI systems in clinical use has their internal threshold set to operate somewhere between 1 and 4 false positives per scan on average ( Van Ginneken et al. 2010 ) . Since pulmonary nodule detection models tend to rely on neural network architectures, they generally do not provide performance guarantees on the sensitivity per scan. To increase trust in AI-based decision support in high-risk domains, valid statistical inference is particularly important. To ensure a patient-centric detection model operates with a high sensitivity per scan, we explore the added benefit of a predictive uncertainty quantification technique from the conformal prediction framework ( Vovk et al. 2005 ; Angelopoulos and Bates 2023 ) , specifically a novel method called conformal risk control (CRC) ( Angelopoulos et al. 2024a ) . The conformal prediction framework relates predictive uncertainty to the size of prediction sets, imitating the human tendency of signalling uncertainty by offering alternatives when unsure ( Cresswell et al. 2024 ) . Conformal methods 2 2 2 Technically, we refer specifically to the popular version of split conformal prediction ( Papadopoulos et al. 2002 ; Lei et al. 2018 ) and its extensions to distribution-free risk control ( Bates et al. 2021 ; Angelopoulos et al. 2022 ; Angelopoulos et al. 2024a ) , but for simplicity we use the umbrella term conformal prediction. wrap around an arbitrary prediction model to turn point predictions into prediction sets with formal statistical guarantees on a particular risk function, such as the sensitivity per scan. Assuming access to a calibration dataset representative 3 3 3 The formal statistical requirement is for future scans to be exchangeable with the calibration data. of future scans, we obtain interpretable statements for non-technical radiologists, such as: “The calibrated detection model is expected to miss at most 10% of ground-truth nodules per scan on average over future scans.” . For the purposes of this case study, CRC amounts to a mathematically principled alternative to FROC analysis. Finally, the task of lung nodule detection in CT scans is consensus-driven, requiring radiologists to agree on the distinction between nodules and non-nodules. Substantial variability exists across radiologists in such a task, even among experienced thoracic radiologists ( Armato III et al. 2009 ) . Disagreement among experts on what constitutes the ground truth concerning a pulmonary nodule is a matter of ontological uncertainty ( Spiegelhalter 2017 ) , measuring the limitations of the proposed prediction pipeline as a description of reality. Ontological uncertainty, as opposed to the perhaps more familiar terms of epistemic and aleatoric uncertainty ( Malinin and Gales 2018 ; Abdar et al. 2021 ; Hüllermeier and Waegeman 2021 ) , is a form of predictive uncertainty often overlooked in the machine learning community. In the context of pulmonary nodule detection, we explore conformal prediction to capture ontological uncertainty by making use of hold-out calibration datasets containing nodules with varying levels of consensus. Insights into this topic are meant to aid healthcare regulators in the recent and ongoing process of designing AI governance structures ( European Commission 2021 ; US Food and Drug Administration 2024 ) by illustrating the risks of using off-the-shelf prediction models in the presence of ontological uncertainty. 1.1 Summary & Outline • We enhance a pulmonary nodule detection model ( Cardoso et al. 2022 ) with conformal risk control (CRC) ( Angelopoulos et al. 2024a ) , leveraging the open-source LIDC-IDRI dataset ( Armato III et al. 2011 ; Armato III et al. 2015 ) . CRC produces prediction sets with the formal guarantee that the sensitivity of a new CT scan is expected at an arbitrary user-specified level (90%). Among nodules annotated by at least three radiologists, our model achieves 91.35% sensitivity at a cost of 2.25 false positives per scan. For reference, this