Fréchet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets
Nicholas Konz, Richard Osuala, Preeti Verma, Yuwen Chen, Hanxue Gu, Haoyu Dong, Yaqian Chen, Andrew Marshall, Lidia Garrucho, Kaisar Kushibar, Daniel M. Lang, Gene S. Kim, Lars J. Grimm, John M. Lewin, James S. Duncan, Julia A. Schnabel, Oliver Diaz, Karim Lekadir, Maciej A. Mazurowski, Full affilia · 2024
Read Fréchet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets by Nicholas Konz, Richard Osuala, Preeti Verma, Yuwen Chen, Hanxue Gu, Haoyu Dong, Yaqian Chen, Andrew Marshall, Lidia Garrucho, Kaisar Kushibar, Daniel M. Lang, Gene S. Kim, Lars J. Grimm, John M. Lewin, James S. Duncan, Julia A. Schnabel, Oliver Diaz, Karim Lekadir, Maciej A. Mazurowski, Full affilia (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01496.txt.
Introduction
Nicholas Konz and a large team of co-authors introduce a new score for comparing sets of medical images in a 2024 paper. Judging whether two image sets come from the same source helps rate image-making models and spot out-of-place data. Scores borrowed from photo work lean on traits that miss body structure, so they can mislead in medicine.
Their score, called FRD, uses standard traits drawn from scans that doctors find clear and useful. Across many tasks, it beat rival scores at spotting odd data, rating image translation, and judging made images. It stayed stable with few images, flagged damage and attacks, and matched how radiologists rated quality. The team also shared code and a broad test frame for such scores.
Catalog record
| CREATOR | Nicholas Konz, Richard Osuala, Preeti Verma, Yuwen Chen, Hanxue Gu, Haoyu Dong, Yaqian Chen, Andrew Marshall, Lidia Garrucho, Kaisar Kushibar, Daniel M. Lang, Gene S. Kim, Lars J. Grimm, John M. Lewin, James S. Duncan, Julia A. Schnabel, Oliver Diaz, Karim Lekadir, Maciej A. Mazurowski, Full affilia |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01496
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fréchet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets Nicholas Konz, Richard Osuala, Preeti Verma, Yuwen Chen, Hanxue Gu, Haoyu Dong, Yaqian Chen, Andrew Marshall, Lidia Garrucho, Kaisar Kushibar, Daniel M. Lang, Gene S. Kim, Lars J. Grimm, John M. Lewin, James S. Duncan, Julia A. Schnabel, Oliver Diaz, Karim Lekadir, Maciej A. Mazurowski, Full affiliations under Author Information . Code: https://github.com/RichardObi/frd-score Evaluation Framework: https://github.com/mazurowski-lab/medical-image-similarity-metrics Abstract Determining whether two sets of images belong to the same or different distributions or domains is a crucial task in modern medical image analysis and deep learning; for example, to evaluate the output quality of image generative models. Currently, metrics used for this task either rely on the (potentially biased) choice of some downstream task, such as segmentation, or adopt task-independent perceptual metrics ( e.g . , Fréchet Inception Distance/FID) from natural imaging, which we show insufficiently capture anatomical features. To this end, we introduce a new perceptual metric tailored for medical images, FRD (Fréchet Radiomic Distance), which utilizes standardized, clinically meaningful, and interpretable image features. We show that FRD is superior to other image distribution metrics for a range of medical imaging applications, including out-of-domain (OOD) detection, the evaluation of image-to-image translation (by correlating more with downstream task performance as well as anatomical consistency and realism), and the evaluation of unconditional image generation. Moreover, FRD offers additional benefits such as stability and computational efficiency at low sample sizes, sensitivity to image corruptions and adversarial attacks, feature interpretability, and correlation with radiologist-perceived image quality. Additionally, we address key gaps in the literature by presenting an extensive framework for the multifaceted evaluation of image similarity metrics in medical imaging—including the first large-scale comparative study of generative models for medical image translation—and release an accessible codebase to facilitate future research. Our results are supported by thorough experiments spanning a variety of datasets, modalities, and downstream tasks, highlighting the broad potential of FRD for medical image analysis. 1 Introduction Comparing image distributions is crucial in deep learning-driven medical image analysis. Example applications include out-of-domain (OOD) detection [ 118 ] , e.g . , for detecting if new medical images were acquired using different protocols; the evaluation of image-to-image translation models, e.g . , for converting MRI (magnetic resonance imaging) to CT (computed tomography) [ 125 , 5 ] ; the quality assessment of images generated to supplement real training data [ 15 , 93 ] ; and others [ 13 ] . However, image distribution metrics from general computer vision ( e.g . , Fréchet Inception Distance/FID [ 47 ] ) often miss key requirements for medical image analysis, and questioning of their unadapted use for this subfield has recently begun [ 86 , 126 , 67 , 127 ] . For example, in medical image OOD detection and image-to-image translation, the focus extends beyond just general image quality to image-level domain adaptation : ensuring that source-domain images ( e.g . , from one scanner, vendor or institution) are compatible with diagnostic models trained on target-domain images from another scanner/vendor/institution, addressing ubiquitous domain shift issues in medical imaging [ 26 , 8 , 123 , 82 , 128 , 73 , 133 , 38 , 129 , 83 ] . Additionally, medical imaging requires metrics that specifically capture anatomical consistency and realism, as well as clinical interpretability [ 76 , 106 , 14 , 112 ] . We argue that these specialized requirements are overlooked by the current metrics used for comparing sets of real and/or synthetic medical images. Figure 1 : Summary of our main contribution: FRD , a metric designed from the ground up for comparing unpaired distributions of real and/or generated medical images. The common approach of comparing medical image distributions in terms of the performance of some downstream task such as segmentation (because direct qualitative image assessment by radiologists is expensive and non-standardized) is driven by the choice of task, and requires costly training and labeling efforts. A task-independent metric that captures general image quality and aligns with expected downstream task performance would therefore be preferable. In computer vision, perceptual metrics like FID are commonly used to evaluate image quality relative to real images [ 47 , 108 , 9 ] , yet these metrics are based on natural image features. Despite this, many applications of medical image translation [ 72 , 123 , 70 , 110 ] and generation [ 93 , 45 , 43 , 114 ] rely on FID (or the related Kernel Inception Distance/KID [ 9 ] ) for evaluation, even though recent findings suggest these metrics may poorly reflect medical image quality [ 66 , 67 , 18 , 127 ] . Our experiments further support this issue. Moreover, to date, no studies have proposed interpretable metrics specifically tailored for comparing unpaired medical image distributions, despite the importance of explainability in medical image analysis. In this paper, we showcase and address limitations in current metrics for comparing unpaired medical image distributions. We begin by evaluating “RadiologyFID” (RadFID) [ 87 ] , a natural extension from FID which uses RadImageNet [ 80 ] features instead of ImageNet [ 22 ] features, that has seen surprisingly little use. We find that RadFID improves upon prior metrics in some areas, yet lacks in interpretability, stability on small datasets, and other essential qualities. To address these gaps, we introduce Fréchet Radiomic Distance ( FRD ), a metric leveraging pre-defined, interpretable radiomic features which are widely used in medical image analysis (see e.g . , Yip and Aerts [132] , Gillies et al. [34] , Van Griethuysen et al. [119] , Lambin et al. [68] ). In addition to the inherent interpretability of the radiomic features it uses [ 20 , 85 , 7 , 130 , 99 ] , FRD offers numerous advantages over learned feature metrics like FID and RadFID, which we demonstrate for various applications via an extensive evaluation framework for medical image distribution similarity metrics, summarized in Fig. 2 . FRD is an improved version of our previous early version “FRD {}_{\text{{v0}}} ” [ 88 ] , being both more robust and capturing a larger, more descriptive space of image features (Sec. 3.1 ); we further verify the resulting improvements in this work with far more extensive intrinsic and extrinsic evaluations than were present in our preliminary study. Figure 2 : Our evaluation framework for FRD and other medical image distribution similarity metrics. We demonstrate these results in key application areas for unpaired medical image distribution comparison, such as out-of-domain (OOD) detection/analysis and the evaluation of image-to-image translation and image generation models. Our experiments span a wide range of medical image datasets and downstream tasks, image translation and generation models, and perceptual metrics. The datasets (see Table 1 ) cover broad medical imaging scenarios that possess different image domains, including breast MRI inter-scanner data from different vendors, inter-sequence brain MRI data involving varying sequences, inter-modality data such as lumbar spine and abdominal MRI and CT, and others, all of which present unique challenges for the explored tasks. We summarize our contributions as follows: 1. We highlight the shortcomings of common metrics for image distribution comparison ( e.g . , FID) in meeting the unique requirements of