Reporting Quality of Research Studies on AI Applications in Medical Images According to the CLAIM Guidelines in a Radiology Journal With a Strong Prominence in Asia Authors: Dong Yeong Kim, Hyun Woo Oh, Chong Hyun Suh Published: 2023 ABSTRACT AbstractObjectiveWe aimed to evaluate the reporting quality of research articles that applied deep learning to medical imaging. Using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines and a journal with prominence in Asia as a sample, we intended to provide an insight into reporting quality in the Asian region and establish a journal-specific audit.Materials and MethodsA total of 38 articles published in the Korean Journal of Radiology between June 2018 and January 2023 were analyzed. The analysis included calculating the percentage of studies that adhered to each CLAIM item and identifying items that were met by ≤ 50% of the studies. The article review was initially conducted independently by two reviewers, and the consensus results were used for the final analysis. We also compared adherence rates to CLAIM before and after December 2020.ResultsOf the 42 items in the CLAIM guidelines, 12 items (29%) were satisfied by ≤ 50% of the included articles. None of the studies reported handling missing data (item #13). Only one study respectively presented the use of de-identification methods (#12), intended sample size (#19), robustness or sensitivity analysis (#30), and full study protocol (#41). Of the studies, 35% reported the selection of data subsets (#10), 40% reported registration information (#40), and 50% measured inter and intrarater variability (#18). No significant changes were observed in the rates of adherence to these 12 items before and after December 2020.ConclusionThe reporting quality of artificial intelligence studies according to CLAIM guidelines, in our study sample, showed room for improvement. We recommend that the authors and reviewers have a solid understanding of the relevant reporting guidelines and ensure that the essential elements are adequately reported when writing and reviewing the manuscripts for publication.Keywords: Reporting quality, Artificial intelligence, Medical imaging, CLAIM guidelines, Asia INTRODUCTION With an increasing number of artificial intelligence (AI) publications in the field of medical imaging, it becomes imperative to have evidence-based guidelines to unify the reporting of AI studies [1,2]. Owing to their complex methodology and weak reporting qualities, AI studies are perceived as challenging for readers [3,4,5]. To address this issue, initial protocols were derived from the standards set for randomized clinical trials. Specifically, the Consolidated Standards of Reporting Trials-AI (CONSORT-AI), Standard Protocol Items: Recommendations for Interventional Trials-AI (SPIRIT-AI) guidelines, and Developmental and Exploratory Clinical Investigations of DEcision support systems driven by AI (DECIDE-AI) were originally developed for consistent reporting of clinical trials and their protocols [6,7,8]. In addition, Standards for Reporting of Diagnostic Accuracy Study-AI (STARD-AI) and Transparent Reporting of a multivariable prediction model of Individual Prognosis Or Diagnosis-AI (TRIPOD-AI) are currently under development [9,10]. Simultaneously, the Checklist for Artificial Intelligence in Medical Imaging (CLAIM), a comprehensive guideline covering the broad application of AI in medical imaging with an emphasis on model development, was published [11] (Table 1). Based on these advantages, CLAIM is one of the best practice guidelines for AI-supported medical imaging research [12] and is endorsed by the Radiological Society of North America (RSNA) journals, which are one of the premier journal groups in medical imaging. Table 1Adherence to CLAIM checklistSection and TopicItem #Checklist itemNumber of articles adheredTitle or Abstract1Identification as a study of AI methodology, specifying the category of technology used (e.g., deep learning)36/38 (95%)2Structured summary of study design, methods, results, and conclusions38/38 (100%)Introduction3Scientific and clinical background, including the intended use and clinical role of the AI approach38/38 (100%)4Study objectives and hypotheses38/38 (100%)MethodsStudy design5Prospective or retrospective study36/38 (95%)6Study goal, such as model creation, exploratory study, feasibility study, non-inferiority trial38/38 (100%)Data7Data sources38/38 (100%)8Eligibility criteria: how, where, and when potentially eligible participants or studies were identified (e.g., symptoms, results from previous tests, inclusion in registry, patient-care setting, location, dates)34/38 (89%)9Data pre-processing steps18/23 (78%)10Selection of data subsets, if applicable8/23 (35%)11Definitions of data elements, with references to Common Data ElementsNA12De-identification methods1/23 (4%)13How missing data were handled0/23 (0%)Ground truth14Definition of ground truth reference standard, in sufficient detail to allow replication26/27 (96%)15Rationale for choosing the reference standard (if alternatives exist)7/10 (70%)16Source of ground-truth annotations; qualifications and preparation of annotators15/20 (75%)17Annotation tools14/18 (78%)18Measurement of inter- and intrarater variability; methods to mitigate variability and/or resolve discrepancies9/18 (50%)Data partitions19Intended sample size and how it was determined1/38 (3%)20How data were assigned to partitions; specify proportions21/23 (91%)21Level at which partitions are disjoint (e.g., image, study, patient, institution)21/23 (91%)Model22Detailed description of model, including inputs, outputs, all intermediate layers and connections20/23 (87%)23Software libraries, frameworks, and packages11/23 (48%)24Initialization of model parameters (e.g., randomization, transfer learning)9/23 (39%)Training25Details of training approach, including data augmentation, hyperparameters, number of models trained14/23 (61%)26Method of selecting the final model5/23 (22%)27Ensembling techniques, if applicable0/23 (0%)Evaluation28Metrics of model performance38/38 (100%)29Statistical measures of significance and uncertainty (e.g., confidence intervals)37/38 (97%)30Robustness or sensitivity analysis1/38 (3%)31Methods for explainability or interpretability (e.g., saliency maps), and how they were validated26/30 (87%)32Validation or testing on external data27/34 (79%)ResultsData33Flow of participants or cases, using a diagram to indicate inclusion and exclusion31/38 (82%)34Demographic and clinical characteristics of cases in each partition26/38 (68%)Model performance35Performance metrics for optimal model(s) on all data partitions38/38 (100%)36Estimates of diagnostic accuracy and their precision (such as 95% confidence intervals)37/38 (97%)37Failure analysis of incorrectly classified cases16/22 (73%)Discussion38Study limitations, including potential bias, statistical uncertainty, and generalizability38/38 (100%)39Implications for practice, including the intended use and/or clinical role38/38 (100%)Other information40Registration number and name of registry2/5 (40%)41Where the full study protocol can be accessed1/38 (3%)42Sources of funding and other support; role of funders34/38 (89%)CLAIM = Checklist for Artificial Intelligence in Medical Imaging, AI = artificial intelligence, NA = not applicable Adherence to CLAIM checklist CLAIM = Checklist for Artificial Intelligence in Medical Imaging, AI = artificial intelligence, NA = not applicable To the best of our knowledge, assessments of the quality of reporting across a broad spectrum of AI medical imaging studies using the CLAIM guidelines as an evaluation tool are limited. Previous studies addressing adherence to CLAIM in AI studies do not fully utilize the advantages of the CLAIM guidelines because they limit the topic to specific disease entities [13,14,15,16,17,18] or specific AI application methods [19,20]. In addition, although several AI reporting guidelines, including CLAIM, are primarily developed in Western countries, AI research is also being actively conducted in Asia, most notably by Chinese and Korean researchers [21]. Therefore, analyzing studies with various applications and disease entities from countries distant from the guidelines’ epicenters could be uniquely informative, for which the Korean Journal of Radiology (KJR) could be a good sample, as it is a broad-spectrum general radiology journal with a reputation and strong presence in contributions from Asia. Moreover, such an analysis would provide a valuable journal-specific audit. This study aimed to evaluate the reporting quality of research articles that have applied deep learning to medical imaging. For this assessment, we used the CLAIM guidelines. We selected a journal with prominence in Asia as a sample, with the intention of providing insight into reporting quality in the Asian region and, also establishing a journal-specific audit. MATERIALS AND METHODS Overview of CLAIM CLAIM is based on the STARD guidelines and has been expanded to cover AI applications in medical imaging, such as classification, image reconstruction, text analysis, and workflow optimization. It focuses on AI model development and particularly emphasizes the generalizability of research [11]. Literature Search Strategy and Study Selection Using the MEDLINE database, we searched for all potential articles discussing AI applications in medical imaging published in a single peer-reviewed journal, the KJR, between June 2018 and January 2023. The search terms were ((“artificial intelligence”) OR (“deep learning”) OR ("machine learning") OR (“convolutional neural network”) OR (“deep neural network”)) AND (“Korean Journal of Radiology” [Journal]). The search date was May 10, 2023. A total of 83 records (i.e., abstracts and titles) were identified from the MEDLINE database, and two reviewers (D.Y.K., with 2 years of experience in radiology, and H.W.O., with 4 years of experience in AI development and research) evaluated the eligibility of each article. Among them the following papers were excluded: 17 review articles, 10 editorials, 5 non-AI studies, and 1 paper each of survey, case report, or erratum. After the first screening, the eligibility of the remaining 48 studies was evaluated. Four records were excluded because they did not use deep learning. Six records were excluded because they were not related to model development or validation studies. As a result, full texts from 38 studies were included in the analysis [22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59] (Fig. 1). Flow diagram of the study selection process. AI = artificial intelligence Data Extraction The following data were independently extracted from the included articles: name of the first author, year of publication, type of AI application (classification, detection, segmentation, or image reconstruction), and study objective (model development or validation). In addition, for each article, we evaluated the sections (title, abstract, introduction, methods, results, discussion, and other information) of the included papers according to CLAIM and referred to the detailed topics (such as study design, data, ground truth, data partitions, and model) of CLAIM to evaluate the methods and results sections. Items #9 to #13 and #20 to #27, which belong to the methods section, were evaluated only in articles on model development. Data were independently extracted by two reviewers (D.Y.K. and H.W.O.). If a disagreement occurred, a third reviewer (C.H.S., with 10 years of experience in performing systematic reviews) was consulted to reach a consensus. Data Analysis We identified the CLAIM checklist items to which ≤ 50% of the articles adhered. We grouped these items and a few additional items (unless included in the “≤ 50% adherence rate” items) we wanted to review further, into 5 relevant domains (Table 2). Suggestions for enhancing the quality of medical imaging articles involving AI were provided based on these established domains. We also compared the rates of adherence to the CLAIM checklist (published in March 2020) between articles published up to December 2020 and those published in January 2021 and later using the chi-squared test. Table 2Grouping of the items with ≤ 50% adherence rate and additional items (#22 and 25)DomainCLAIM itemsChecklist1.Information concerning data and data partitioning#10, #12, #13, #191.Report detailed information about data and data partitions.- If subsets of the raw data were used, report detailed information.- Report the methods of de-identification.- State clearly how missing data were handled.2.Mention the intended sample size and a reference about how to calculate sample size.2.#183.Provide methods for evaluating inter- and intrarater variability and ways to resolve them.3.Details concerning model and training#22, #23, #24, #25, #26, #274.Provide techniques to construct and train AI model for reproducibility and transparency.- Report a detailed structure of the model and the name of software libraries.- Indicate parameter initialization methods.- Describe all the training procedures and hyperparameters.- State model selection method and ensemble method, if applicable.4.Evaluating model performance#305.Perform robustness or sensitivity analysis to ensure software to keep an “acceptable” behavior, in spite of exceptional or unforeseen execution conditions.5.Other information#40, #416.If the current study is a prospective study, it is recommended to provide registration information.7.Share all computer code used for modeling or data analysis in a publicly accessible repository.CLAIM = Checklist for Artificial Intelligence in Medical Imaging, AI = artificial intelligence Grouping of the items with ≤ 50% adherence rate and additional items (#22 and 25) CLAIM = Checklist for Artificial Intelligence in Medical Imaging, AI = artificial intelligence RESULTS Characteristics of the Included Studies The characteristics of the 38 included studies are summarized in Supplementary Table 1. In terms of the type of AI application, 9 studies (24%) applied AI for classification [22,27,31,32,38,39,43,55,57], 6 studies (16%) for detection [35,36,49,50,51,58], 11 studies (29%) for segmentation [25,28,29,33,37,40,41,45,46,48,53], and 12 studies (31%) for image reconstruction [23,24,26,30,34,42,44,47,52,54,56,59]. The study objective of 23 papers (61%) was classified as model development [22,23,25,26,28,29,30,31,32,35,36,37,38,39,42,43,45,46,48,49,53,55,59], and the remaining 15 papers (39%) were classified as validation studies [24,27,33,34,40,41,44,47,50,51,52,54,56,57,58]. Five studies (13%) were prospective [22,41,42,55,58] and 33 studies (87%) were retrospective [23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,43,44,45,46,47,48,49,50,51,52,53,54,56,57,59]. Adherence to CLAIM The included articles were evaluated for adherence to each item of the CLAIM checklist (Table 1). Of the 42 guideline items, 12 items (29%) were reported in ≤ 50% of the articles (Fig. 2). Most of the studies met the criteria for the title, abstract, introduction, and discussion sections, but there were frequent instances of incomplete reporting in the methods and other information sections, particularly concerning the data topic. None of the studies reported handling missing data (item #13). Only one study respectively reported the use of de-identification methods (item #12), intended sample size (item #19), robustness or sensitivity analysis (item #30), and the full study protocol (item #41). Thirty five percent of the studies reported the selection of data subsets (item #10), 40% reported registration information (item #40), and 50% measured inter and intrarater variability (item #18). Bar chart demonstrating the proportion of articles that satisfied each item of the CLAIM checklist. Twelve items (29%) were reported in ≤ 50% of the articles. CLAIM = Checklist for Artificial Intelligence in Medical Imaging For the 12 items that were met by less than 50% of the included articles, there were no significant changes in the adherence rates between the two periods. There was a slight increase in adherence to data partitions, model, and training-related items (#20, #21, and #22) in the methods section, but without a statistical difference, while item #25, related to detailed reporting on the training procedure, showed a statistically significant improvement (33% to 79%; P = 0.031). In the results section, item #34, which was regarding reporting demographic and clinical characteristics for each data partition, also showed marginal improvement (45% to 78%; P = 0.05). We grouped the 12 items with ≤ 50% adherence rate and two additional items (#22 and #25) that we wanted to emphasize, into 5 domains, as listed below (Table 2). Information Concerning Data and Data Partitioning (Items #10, #12, #13, and #19) The use of data subsets, such as data cropping, focusing on a specific segment of the dataset, could facilitate model training and testing [60]. According to the CLAIM guidelines, the use of data subsets should be indicated when applicable. It was observed that 35% of the studies employed this method in their research [22,26,31,36,39,43,48,55]. De-identification is an important ethical aspect in AI research. In our review, only one study (4%) explicitly reported the use of anonymization [31]. In the reviewed articles, the target tasks were primarily related to computer vision such as image classification, image segmentation, detection, and image recognition. Owing to the nature of these tasks, there was no mention of techniques for handling missing data, which are more commonly associated with tabular data. Item #19 addresses the intended sample size. Only one article (3%) met these criteria [51]. It mentioned the intended sample size and provided reference for calculating the sample size tables for receiver operating characteristic studies. Concrete Descriptions about Ground Truth (Item #18) Item #18 concerns measuring inter and intrarater variability and the method to resolve it. This could not be applied to 20 papers. Therefore, item #18 was evaluated for 18 papers altogether, and 50% of these articles met the criteria [32,38,39,44,45,46,48,49,58]. Two studies [32,39] did not suggest methods to assess inter and intrarater variability, whereas seven studies [38,44,45,46,48,49,58] did not report methods to reduce or mitigate this variability or resolve discrepancies. Details Concerning Model and Training (Items #22, #23, #24, #25, #26, #27) In the quest for reproducibility and transparency in the field of AI research, a comprehensive and detailed description of an AI model’s structure is a critical element. CLAIM requests a ‘complete detailed structure’: the components of input and output, the structure of the neural network including pooling, normalization, regularization, and activation layer. It was found that 20 out of 23 (87%) articles provided a detailed structure of their proposed model [22,23,25,26,28,29,32,35,36,37,39,42,43,45,46,48,49,53,55,59]. Among the three articles that did not provide enough details, one cited a previous paper for its model structure [30], whereas the other two articles lacked sufficient detail. Software libraries (item #23), initialization of model parameters (item #24), details of the training approach (item #25), method of selecting the final model (item #26), and ensembling techniques (item #27) are described in the Supplementary Material. Evaluating Model Performance (Item #30) Item #30 concerns robustness or sensitivity analysis. Among the included articles, only one (3%) [25] satisfied the criteria. The paper mentioned that subgroups of various clinical conditions were included, and several types of computed tomography scanners were used to develop a robust deep-learning algorithm. Other Information (Items #40 and #41) Item #40 is related to clinical trial registration, but most of the included studies were retrospective. Therefore, only five studies could be evaluated, and 40% of these studies reported registration information [41,58]. CLAIM emphasizes that authors should share all computer code used for modeling or data analysis in a publicly accessible repository; in this aspect, item #41 was satisfied by only one article (3%) [53]. Another study [56] mentioned that all data generated or analyzed were included in the text and supplements, but the computer code was not publicly disclosed. DISCUSSION Our study revealed the focus areas for improving the reporting quality of studies on AI applications in medical imaging. Although the results were obtained from a single journal, given the status of the journal (including Q1 status according to the Journal Citation Reports™ and Scimago Journal & Country Rank [61,62]), they may serve as a snapshot of the reporting quality among articles generally regarded as high-quality research studies in the field of radiology. Our evaluations can be segmented into five categories, where detailed guidance and recommendations can be provided: 1) information concerning data and data partitioning, 2) concrete descriptions about ground truth, 3) details concerning model and training, 4) evaluating model performance, and 5) other information. The results of this study were similar to those of previous studies using CLAIM [13,15,16]. In the case of item #18, dealing with inter and intrareader variability, only 50% of the studies satisfied the criteria, and this was also low in previous studies [13,16]. Human perception remains in the initial stage of image reading; however, a radiologist’s proficiency depends on multiple factors. Consequently, the outcomes of an imaging technique frequently hinge on the inherent qualities of the observer. To address this issue, it is advised to involve multiple observers and conduct independent readings to gain a comprehensive understanding of potential variations in the results [63]. In addition, the achievement rate of item #30, robustness, or sensitivity analysis, was very low, and similar results have been reported in previous studies [15,16]. Robustness can be defined as the ability of the software to maintain “acceptable” behavior despite exceptional or unforeseen execution conditions [64]. To some extent, the achievements of deep learning models rely on their ability to generalize and remain stable. Studies have demonstrated that these models can produce different outputs when presented with slight variations in input data. Such response variability to minor changes might indicate algorithmic instability, potentially resulting in misclassification and challenges in generalization [65]. Therefore, it is important to evaluate the robustness and stability of AI models before their clinical implementation, especially in the field of medical imaging, and authors should consider reporting them. In the case of the ground truth topic in the methods section, the report rate exceeded 50%, except for item #18, which had a different result from that in previous studies that showed a low report rate for this item [13,15,16]. While previous studies collected and evaluated specific disease imaging studies, the current study conducted a wide range of evaluations without distinguishing between AI applications and diseases. As a result, many papers dealing with image reconstruction and segmentation were included. Since this is a research area where alternative reference standards are relatively difficult to find, it is judged that the report rate was high because this study was a little less stringent in defining the ground truth than other studies. Reproducibility and transparency in deep-learning modeling are crucial factors for enhancing the quality of research, and standardized guidelines are necessary to achieve them. However, in the current guidelines, items #22 and #25, may be perceived as vague or overly strict. For example, if a convolution layer is employed as an intermediate layer, many details, such as stride, padding, dilation rate, and bias should be applied [66]. Describing these details thoroughly may be perceived as being redundant or overly strict. It would be advisable to provide further guidelines that specify the methodological details that should be included rather than demanding a full, exhaustive description. Furthermore, the clarity can be enhanced by reorganizing certain items. For instance, item #10, which is related to the preprocessing of the input, could be included in item #22. Part of item #25 concerning the selection method of the best-performing model, could be transferred to item #26. Finally, item #13, the handling of missing data, pertains only to tabular data [67]. Therefore, a modification that mandates this item only in papers using tabular data can enhance the clarity of the guidelines. As mentioned above, there are many opinions regarding improvements through the reorganization or clarification of some items, and an updated CLAIM that reflects these is being developed [12]. Sharing executable algorithms or data in a publicly accessible repository is currently recommended for publication by most peer-reviewed journals, including Radiology [68] and not just by the CLAIM guidelines. However, a previous study using CLAIM [16] reported a low adherence rate to this policy. Code sharing facilitates the evaluation of an AI algorithm using data from the intended healthcare system, which is required to confirm the algorithm’s generalizability to the user’s environment [2,69]. In addition, code sharing can provide users with a deeper insight into the necessary computing power and logistical factors, such as data transfer and image preprocessing [2]. Our study has several limitations. First, we used a specific journal as the study sample; therefore, our results may have limited generalizability and should be viewed along with other similar published studies. Second, CLAIM comprises a multitude of reporting guidelines for AI research studies and has its own limitations. For example, the direction of reporting the sample size does not distinguish between the training and test datasets. While sample size estimation is critical for testing an algorithm with adequate statistical power, a priori estimation of an adequate training data size is not entirely practical or feasible [2]. CLAIM is currently under revision [12], and our study results would need to be updated when new reporting guidelines emerge. Finally, there is potential inexperience in our analysis owing to the inaugural application of the CLAIM criteria. Unfamiliarity with certain aspects of the criteria may have influenced our evaluation. In conclusion, the reporting quality of AI studies with respect to CLAIM of AI studies in our study sample, showed room for improvement. We recommend that the authors and reviewers have a solid understanding of the relevant reporting guidelines and ensure that the essential elements are adequately reported when writing and reviewing the manuscripts for publication. Footnotes Conflicts of Interest: Chong Hyun Suh, the Assistant to the Editor of the Korean Journal of Radiology, was not involved in the editorial evaluation or decision to publish this article. All authors have declared no conflicts of interest. Author Contributions: Conceptualization: all authors.Data curation: Dong Yeong Kim, Hyun Woo Oh.Formal analysis: Dong Yeong Kim, Hyun Woo Oh.Funding acquisition: Chong Hyun Suh.Investigation: Chong Hyun Suh.Methodology: Chong Hyun Suh.Project administration: Chong Hyun Suh.Resources: Chong Hyun Suh.Software: Dong Yeong Kim, Hyun Woo Oh.Supervision: Chong Hyun Suh.Validation: Chong Hyun Suh.Visualization: Dong Yeong Kim, Hyun Woo Oh.Writing—original draft: Dong Yeong Kim, Hyun Woo Oh.Writing—review & editing: all authors. Conceptualization: all authors. Conceptualization: all authors. Data curation: Dong Yeong Kim, Hyun Woo Oh. Data curation: Dong Yeong Kim, Hyun Woo Oh. Formal analysis: Dong Yeong Kim, Hyun Woo Oh. Formal analysis: Dong Yeong Kim, Hyun Woo Oh. Funding acquisition: Chong Hyun Suh. Funding acquisition: Chong Hyun Suh. Investigation: Chong Hyun Suh. Investigation: Chong Hyun Suh. Methodology: Chong Hyun Suh. Methodology: Chong Hyun Suh. Project administration: Chong Hyun Suh. Project administration: Chong Hyun Suh. Resources: Chong Hyun Suh. Resources: Chong Hyun Suh. Software: Dong Yeong Kim, Hyun Woo Oh. Software: Dong Yeong Kim, Hyun Woo Oh. Supervision: Chong Hyun Suh. Supervision: Chong Hyun Suh. Validation: Chong Hyun Suh. Validation: Chong Hyun Suh. Visualization: Dong Yeong Kim, Hyun Woo Oh. Visualization: Dong Yeong Kim, Hyun Woo Oh. Writing—original draft: Dong Yeong Kim, Hyun Woo Oh. Writing—original draft: Dong Yeong Kim, Hyun Woo Oh. Writing—review & editing: all authors. Writing—review & editing: all authors. Funding Statement: This work was supported by the National Research Foundation of Korea (NRF-2021R1C1C1014413). Availability of Data and Material The datasets generated or analyzed during the study are available from the corresponding author on reasonable request. Supplement The Supplement is available with this article at https://doi.org/10.3348/kjr.2023.1027. References Associated Data Supplementary Materials Data Availability Statement The datasets generated or analyzed during the study are available from the corresponding author on reasonable request.