Update report on the quality of gliomas radiomics: An integration of bibliometric and radiomics quality score Authors: Xu Cao, Ming Xiong, Zhi Liu, Jing Yang, Yu-Bo Kan, Li-Qiang Zhang, Yan-Hui Liu, Ming-Guo Xie, Xiao-Fei Hu Published: 2024 ABSTRACT BACKGROUNDDespite the increasing number of publications on glioma radiomics, challenges persist in clinical translation.AIMTo assess the development and reporting quality of radiomics in brain gliomas since 2019.METHODSA bibliometric analysis was conducted to reveal trends in brain glioma radiomics research. The Radiomics Quality Score (RQS), a metric for evaluating the quality of radiomics studies, was applied to assess the quality of adult-type diffuse glioma studies published since 2019. The total RQS score and the basic adherence rate for each item were calculated. Subgroup analysis by journal type and research objective was performed, correlating the total RQS score with journal impact factors.RESULTSThe radiomics research in glioma was initiated in 2011 and has witnessed a surge since 2019. Among the 260 original studies, the median RQS score was 11, correlating with a basic compliance rate of 46.8%. Subgroup analysis revealed significant differences in domain 1 and its subitems (multiple segmentations) across journal types (P = 0.039 and P = 0.03, respectively). The Spearman correlation coefficients indicated that the total RQS score had a negative correlation with the Journal Citation Report category (-0.69) and a positive correlation with the five-year impact factors (0.318) of journals.CONCLUSIONGlioma radiomics research quality has improved since 2019 but necessitates further advancement with higher publication standards. Core Tip: Bibliometric and radiomics quality scores were used to assess glioma radiomics research, and the results showed the quality of glioma radiomics research and reporting requires further improvement. INTRODUCTION Gliomas are the most common primary tumors of the central nervous system. While preoperative non-invasive imaging can provide critical information on differential diagnosis, molecular typing, grading, prognosis, and treatment response of gliomas, conventional analysis methods often underperform in delivering adequate results[1]. Radiomics, an artificial intelligence (AI)-assisted technology, offers a novel approach by extracting high-dimensional feature data from medical images[2]. Therefore, radiomics has attracted great interest among clinicians and researchers[1], with its application in glioma studies including differential diagnosis[3,4], grading and classification[5-7], survival analysis[8,9], and treatment response[10]. Despite its potential, radiomics is not yet widely adopted in clinical practice, primarily due to the lack of large-scale data validation, methodological rigor, and high-quality reporting. Radiomics Quality Score (RQS), an evaluation system developed by Lambin, aims to address these challenges by assessing the effectiveness and integrity of radiomics research[11]. Based on 16 criteria, the RQS rewards and penalizes methodological and analytical aspects to promote best scientific practices. Previous studies have utilized the RQS to evaluate the quality of radiomics in neuro-oncology, revealing a notable absence of high-level evidence[12]. Our present study aims to update the insights into the field of glioma radiomics by focusing on articles published since February 2019, which were not covered in prior assessments[13]. Bibliometrics, as a method of information visualization, enables a quantitative analysis of publication characteristics such as timing, authorship, and citations, providing a comprehensive view of research trends and hotspots within a specific domain[14-16]. It can achieve quantitative analysis of literature in a specific research field in a worldwide context through statistical methods and visualizing the results with the helps of software such as VOSviewer and CiteSpace, which can analyze and visualize literature data, general network data, and text data through co-occurrence clustering to present the scientific knowledge map[14-16]. In this study, we apply bibliometric analysis to delineate the development and trends in glioma radiomics research and employ the RQS to evaluate the quality of recent studies on adult-type diffuse gliomas. Our findings aim to offer valuable guidance for future research endeavors and facilitate clinical translation in the field of glioma radiomics. MATERIALS AND METHODS Literature retrieval and manual screening We designed a search strategy tailored to our research focus on radiomic applications in glioma, with specific inclusion criteria including: (1) English-language papers; (2) Focused on glioma; (3) Integrated radiomic analysis; and (4) Published between 2011 and 2023. The final search was updated on April 22, 2023. A team of three authors (Cao X, Xiong M, and Yang J) screened potentially relevant papers, resolving any uncertainties through discussion. Unlike systematic review, bibliometric analysis primarily requires abstract screening, with full-text review reserved for select cases. According to the screening criteria, a total of 773 articles were included for bibliometric analysis in the study after retrieval from PubMed and Embase databases and integration with the WOS database (the literature retrieval method can be found in Supplementary material). Then two authors (Liu Z and Zhang LQ) further screened out articles that could be used for RQS scoring. Articles that met the following criteria were excluded: (1) Review or Meta-analysis; (2) Original research articles published in 2019 and before; (3) Focused on localized glioma and pediatric and diffuse midline gliomas, and (4) Radiomics or radiogenomics studies using only correlations and without a performance measurement. Figure 1 shows the inclusion and exclusion criteria of bibliometric and RQS analyses in this study. Flow chart of inclusion and exclusion criteria for bibliometrics and Radiomics Quality Score analyses. Bibliometric analysis and visualization We employed VOSviewer (version 1.6.18) and CiteSpace (version 6.2.R4) for bibliometric visualization, examining cooperation networks and keyword co-occurrences[17-20]. Scimago Graphica (version 1.0.35) supported the visualization of country/region analysis[21,22]. CiteSpace was utilized to assess leading journals' collaboration and field-specific interrelationships. Microsoft Excel 2019 facilitated the analysis of targeted files and publication trends. The top ten most cited or productive entities in each category were analyzed and tabulated. The top ten most cited or productive countries/regions, institutions, and journals were also analyzed. RQS data extraction and analysis Three experienced reviewers (Yang J, Kan YB, and Cao X, with 5, 8, and 10 years in neuro-oncology imaging, respectively) scored RQS based on full-text articles. A workshop was conducted to ensure a unified understanding of RQS assessment criteria. Data extraction was guided by six predefined domains, with details of the 16 RQS components provided in Supplementary Table 1 in the Supplementary material. Each article was independently evaluated by two assessors, with discrepancies resolved through consensus discussions moderated by Cao X. Statistical analysis Basic adherence for RQS items, domains, or total scores was set at a minimum score of 1. Adherence rates were calculated as ratios of compliant articles. The Kruskal-Wallis test was applied to compare RQS scores across research types (differential diagnosis, grading or molecular typing, prognosis, and treatment response) and journals (imaging, clinical and comprehensive journals) subgroups, with Bonferroni correction for multiple comparisons. Statistical significance was set at P < 0.05. All analyses were performed using SPSS software package (version 22, SPSS Inc., Chicago, Illinois). RESULTS Annual publication trends Our bibliometric analysis encompassed 773 papers, including 667 original articles and 106 review articles (Figure 1). The inception of glioma radiomics research in 2011 marked a gradual increase, with a significant surge since 2019, constituting nearly 80% of the total decade's output (Figure 2). The collective citation count stands at 15525, with an average of 20.71 citations per paper. Global trend of publications on glioma radiomics from 2011 to 2023. Productive countries/regions Fifty-three countries/regions contributed to the field, with the United States (274 publications), China (242), and Germany (73) leading in publication volume, representing 35.45%, 31.31%, and 9.44% of the total, respectively. The United States also dominated in total citations and average citations per paper, at 7959 and 29.05, respectively, surpassing China's 4039 and 16.70 citations (Supplementary Table 2). The cooperative network among countries could be divided into six cooperative network clusters by VOS clustering and showed that the United States and China had more cooperative relations with other countries (Figure 3). Inter-country cooperation network map. The size of the nodes indicates the amount of publications, and the thickness of the lines represents the intensity of cooperation. Productive institutions Over 12 years, 200 institutions have published on this topic, with the Chinese Academy of Sciences, University of Pennsylvania, and Capital Medical University emerging as the top three contributors. The German Cancer Research Center excelled in citation metrics, suggesting high impact (Supplementary Table 3). In addition, Stanford University, Chinese Academy of Sciences, and University of Utah showed the strongest institutional linkages, and the transnational cooperation among institutions of other countries was generally low (Supplementary Figure 1). Journals The research was disseminated across 197 journals, with Neuro-Oncology, Frontiers in Oncology, and Cancers being the most prolific (Table 1). The top 10 journals in this domain were ranked above the journal citation report (JCR) Q2 division. Scientific Reports, European Radiology, and Neuro-Oncology garnered the highest citations, with counts of 1483, 920, and 906, respectively. The co-occurrence network of these journals is depicted in Figure 4. Co-occurrence network analysis of top journals. The lines represent partnerships, and different colors represent different years of cooperation. Larger circles mean more publications. Table 1Top 10 journals related to gliomas radiomics Rank Journal (country) Count IF (2022) JCR (2022) Total citations 1 Neuro-Oncology (United States)6315.9Q19062 Frontiers in Oncology (Switzerland)624.7Q23723Cancers (United States)465.2Q21904 European Radiology (Germany)445.9Q19205 Scientific Reports (England)414.6Q214836 Journal of Magnetic Resonance Imaging (United States)204.4Q15327 American Journal of Neuroradiology (United States)193.5Q24838 Medical Physics (United States)183.8Q21029 Journal of Neuro-Oncology (United States) 173.9Q228710 International Journal of Radiation Oncology Biology Physics (United States)157Q171IF: Impact factor; JCR: Journal citation report. Top 10 journals related to gliomas radiomics IF: Impact factor; JCR: Journal citation report. Keywords In the co-occurrence analysis of keywords, key emerging terms in recent years included "glioblastoma", "radiomics", "survival", "classification", and "MRI" (Figure 5). Disease type and clinical goal analyses reveal a focus on classification and survival, with less emphasis on prognosis and diagnosis (Figure 6A). "Glioblastoma" and "astrocytoma" dominate the study subjects (Figure 6B). Co-occurrence network analysis of keywords regarding radiomics in glioma. Analysis of keywords from clinical goals and disease types. A: Distribution of publications by goal; B: Distribution of publications by disease category. RQS assessment characteristics The RQS assessment included 260 articles, with 20.4% lacking verification and 27.3% being multicenter studies. Detailed characteristics and RQS scores are available in Supplementary Table 4. Research goals were categorized into four subgroups including differential diagnosis (15.7%), treatment response (12.3%), prognosis prediction (33.5%), and grading or molecular typing (38.5%) with the dataset sizes (median, interquartile range) being (120.20-935), (90.15-463), (162.22-704), and (130.36-894), respectively (P < 0.01). The overall basic adherence rate for RQS was 46.8%, with rates for domains 1 to 6 being 99.2%, 79.6%, 77.7%, 98.5%, 1.5%, and 58.1%, respectively (Table 2). Table 2Basic adherence rate according to the six key domains Radiomics quality score Basic adherence rate (%) Total 16 items46.80Domain 1: Protocol quality and stability in image and segmentation99.20Protocol quality98.80Test-retest36.90Phantom study10.40Multiple segmentation33.10Domain 2: Feature selection and validation79.60Feature reduction or adjustment of multiple testing98.80Validation79.6Domain 3: Biologic/clinical validation and utility77.70Multivariate analysis with non-radiomics features67.30Biologic correlates27.70Comparison to “gold standard”5.80Potential clinical utility 118.10Domain 4: Model performance index98.50Discrimination statistics97.30Calibration statistics60.80Cut-off analysis54.60Domain 5: High level of evidence1.50Prospective study0.40Cost-effective analysis1.10Domain 6: Open science and data58.10 Basic adherence rate according to the six key domains Subgroups The median and interquartile range of total RQS score was 11 (9-14). Subgroup analysis by journal type revealed significant differences in domain 1 (P = 0.039) and its subitems (P = 0.03). Pairwise comparisons between groups showed significant differences only in domain 1 (P = 0.036). Research goal subgroup analysis indicated significant differences across all domains except domain 3 (P = 0.18). Specific differences are detailed in Table 3 and Table 4, with pairwise comparisons provided in Supplementary Table 5. Table 3Subgroup analysis in journal types, median (interquartile range) Radiomics quality score Median score Imaging journals Clinical journals Comprehensive journals P value Total 36 points11 (9–14)11 (8.25–14)12 (9–16)11.5 (8–15)0.379Domain 1: Protocol quality and stability in image and segmentation (0 to 5 points)1 (1–2)1 (1–2)2 (1–2)1 (1–2)0.039Image protocol quality (2)1 (1–1)1 (1–1)1 (1–1)1 (1–1)0.201Multiple segmentations (1)0 (0–1)0 (0–1)0 (0–1)0 (0–0.25)0.03Phantom study on all scanners (1)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.205Imaging at multiple time points (1)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.383Domain 2: Feature selection and validation (-8 to 8 points)5 (5–6)5 (5–6)5 (5–6)5 (1–2)0.553Feature reduction or adjustment for multiple testing (-3 or 3)3 (3–3)3 (3–3)3 (3–3)3 (3–3)0.539Validation (-5, 2, 3, 4, or 5)2 (2–3)2 (2–3)2 (2–3)2 (1.5–2.25)0.585Domain 3: Biologic/clinical validation and utility (0 to 6points)1 (1–2)1 (0.25–2)1 (1–2)1 (0–2)0.613Multivariable analysis with non-radiomics features (1)1 (0–1)1 (0–1)1 (0–1)1 (0–1)0.059Detect and discuss biological correlates (1)0 (0–1)0 (0–0)0 (0–1)0 (0–1)0.075Comparison to ‘gold standard’ (2)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.669Potential clinical utility (2)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.628Domain 4: Model performance index (0 to 5 points)3 (2–4)3 (2–4)3 (2–4)3 (2–5)0.315Cut-off analyses (1)1 (0–1)0 (0–1)1 (0–1)1 (0–1)0.101Discrimination statistics (2)2 (1–2)2 (1.5–2)2 (1–2)2 (1.75–2)0.071Calibration statistics (2)1 (0–2)1 (0–2)1 (0–2)0.5 (0–2)0.175Domain 5: High level of evidence (0 to 8 points)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.588Prospective study registered in a trial database (7)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.435Cost-effectiveness analysis (1)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.236Domain 6: Open science and data (0 to 4 points)0.653 (0–1)1 (0–1)1 (0–1)0.5 (0–2)0.649 Subgroup analysis in journal types, median (interquartile range) Table 4Subgroup analysis in the goal of research, median (interquartile range) Radiomics quality score Differential diagnosis Treatment response Prognosis prediction Grading or molecular typing P value Total 36 points10 (8–13)11 (8–14)13 (11–15)11 (9–15)0Domain 1: Protocol quality and stability in image and segmentation (0 to 5 points)2 (1–2)2 (1–2)1 (1–2)1 (1–2)0.006Image protocol quality (2)1 (1–1)1 (1–1)1 (1–1)1 (1–1)< 0.001Multiple segmentations (1)1 (1–1)0 (0–1)0 (0–0.75)1 (0–1)< 0.001Phantom study on all scanners (1)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.002Imaging at multiple time points (1)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.36Domain 2: Feature selection and validation (-8 to 8 points)5 (–2–5)5 (–2–6)5 (5–6)5 (5–5)0.012Feature reduction or adjustment for multiple testing (-3 or 3)3 (3–3)3 (3–3)3 (3–3)3 (3–3)0.601Validation (-5, 2, 3, 4, or 5)2 (–5–2)2 (2–2)2 (2–3)2 (–5–3)0.007Domain 3: Biologic/clinical validation and utility (0 to 6 points)2 (0–2)1 (1–2)1 (1–2)2 (0–2)0.18Multivariable analysis with non-radiomics features (1)0 (0–0)1 (0–1)1 (1–1)1 (0–1)< 0.001Detect and discuss biological correlates (1)0 (0–0)0 (0–0)0 (–0–1)0 (0–1)0.001Comparison to ‘gold standard’ (2)0 (0–2)0 (0–0)0 (0–0)0 (0–0)< 0.001Potential clinical utility (2)0 (0–2)0 (0–0)0 (0–0)0 (0–0)0.032Domain 4: Model performance index (0 to 5 points)2 (2–2)3 (2–4)4 (3–5)3 (2–4)< 0.001Cut-off analyses (1)0 (0–1)0 (0–1)1 (1–1)0 (0–1)< 0.001Discrimination statistics (2)2 (1.5–2)2 (2–2)2 (1–2)2 (1–2)0.049Calibration statistics (2)0 (0–0)0 (0–2)0 (1–2)0 (0–2)< 0.001Domain 5: High level of evidence (0 to 8 points)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.001Prospective study registered in a trial database (7)0 (0–0)0 (0–0)0 (0–0)0 (0–0)0.148Cost-effectiveness analysis (1)0 (0–0)0 (0–0)0 (0–0)0 (0–0)< 0.001Domain 6: Open science and data (0 to 4 points)0 (0–1)1 (0–1)1 (0–1)0 (0–1)0.016 Subgroup analysis in the goal of research, median (interquartile range) Correlation between RQS and bibliometrics The Spearman correlation coefficients between the total RQS score and the JCR division and the 5-year impact factors were -0.69 and 0.318, respectively, indicating a stronger correlation with the JCR division. The total RQS score positively correlated with sample size (coefficient 0.336) but was not significantly associated with research goals, journal types, or publication year (P = 0.523, P = 0.349, and P = 0.435, respectively). DISCUSSION Our bibliometric analysis identified 2019 as a critical inflection point, marking a rapid increase in glioma radiomics research. Prominent contributions from the United States, China, and Germany, alongside a predominance of American publishers, highlight the global significance of this field. The presence of both medical and engineering journals among the top ten indicates a robust interdisciplinary approach, underscoring the importance of integrating medical and engineering perspectives in radiomics research. Furthermore, the analysis reveals a need to enhance transnational inter-agency cooperation, potentially offering more profound insights into glioma research globally. The introduction of molecular biomarkers into the classification of gliomas since 2016[23], with increased emphasis in 2021[24,25], aligns with the observed focus on classification in most studies. Glioblastoma, the glioma subtype with the worst prognosis and highest postoperative recurrence rate, understandably commands significant research attention[26], and the trend of similar proportions is presented in keyword analysis (Figure 6B). Thus, our RQS analysis focused on adult-type diffuse gliomas. Our study showed an increase in ROS score compared to a 2019 survey[12] (37.1% vs 46.8%), which may be attributed to enhancements in several key areas including Domain 1 (which assesses the quality of image acquisition and preprocessing) and Domains 3 and 4 (which focus on the validation and performance of radiomics models). Notably, the openness and sharing of scientific data in Domain 6 have seen the most substantial increase, reflecting a positive trend towards transparent and reproducible research practices. Despite these improvements, the overall score of Domain 2 remained stagnant, although its validation score did increase, indicating a need for continued focus on this area. However, several obstacles persist in clinical practice, warranting further attention. In the realm of test-retest reliability within Domain 1, an alarmingly low number of studies—specifically two[27,28]-have employed multi-timepoint imaging. The impact of varying imaging protocols on feature robustness is another area that requires greater scrutiny, as evidenced by the mere 10.4% of studies incorporating phantom-based validation. While standard image preprocessing pipelines are essential for harmonizing captured images[29], the adoption of tools like Combat presents a significant advancement. Combat has proven effective not only in reducing inconsistencies in radiomic feature values across magnetic resonance imaging scanners and centers but also in refining the precision of subsequent experimental analyses[30-32]. Although the validation score has improved compared to previous studies, there is a notable absence of generalizability. External validation is imperative, as algorithm performance is often compromised when assessed on external datasets[30,31]. Our analysis indicated a positive correlation between the RQS total score and sample size, suggesting that larger sample sizes may enhance the reliability of external validation. The disparity in RQS scores across different research goals could be attributed to the limited sample size in the treatment response subgroup, which adversely affects external validation scores. The lack of prospective studies, which carry the highest weight in scoring, is another critical area for future research. The absence of cost-benefit analyses in the studies surveyed may further impede the clinical translation and application of radiomics. The significant improvement in the "Open science and data" domain is a step in the right direction; however, achieving complete re-presentation of research remains elusive. The reluctance to share open-access data may stem from restrictions on patient data release. Nonetheless, the full disclosure of scientific data, devoid of sensitive information and with accessible storage solutions, is essential for the clinical translation of radiomic findings[32,33]. These challenges are not confined to brain tumor research. A systematic review of prostate studies[34] revealed no public data sharing, and similar adherence issues were found in multiple myeloma studies[35] and rectal cancer research[36]. The radiomics field in stroke[37] also faces the dilemma of repeatability. While not all radiomics studies require phantom-based test-retest experiments or multi-timepoint scanning, a prospective design, multi-center collaboration, adequate sample size, comprehensive external verification, and a focus on retest research are pivotal for the future of glioma radiomics. Our findings suggest a correlation between the reputation of journals and the publication of high-quality research, with a Spelman correlation coefficient of -0.69 between the RQS total score and the JCR division. This underscores the critical role of publishers and reviewers in enforcing stringent publication standards and addressing the aforementioned obstacles in clinical transformation. The recent Check List for Evaluation of Radiomics (CLEAR checklist) published by the European Society of Radiology and the European Society of Medical Imaging Informatics offers a valuable framework for enhancing the quality of radiomics research reporting[38]. Our study acknowledges several limitations. The quality assessment was primarily concentrated on adult-type diffuse gliomas, which, while prevalent, do not encompass the full spectrum of gliomas. Additionally, our evaluation did not extend to studies incorporating deep learning or its integration with radiomics, as the RQS is not optimally designed to assess such methodologies. Nevertheless, emerging AI modeling checklists, including the Minimum Information for Medical AI Reporting checklist[39], Checklist for AI in Medical Imaging[40], and extensions of the Transparent Reporting of a multivariable prediction model of Individual Prognosis Or Diagnosis statement and the Prediction model Risk Of Bias Assessment Tool[41], offer complementary criteria to enhance the RQS framework. Notably, a novel methodological scoring tool, Metrics[42], developed by an international consortium of experts, addresses this gap, providing a comprehensive 30-item evaluation spanning nine domains, suitable for both handcrafted radiomics and deep learning-based pipelines. Future comparisons between these tools and the exploration of Metrics' impact on enhancing imaging research and clinical translation will be pivotal. CONCLUSION In summary, enhancing the methodological rigor and scientific reporting of glioma radiomics is imperative. The integrative approach of bibliometrics with RQS offers significant reference value for the future, guiding research design, execution, reporting, and pre-publication peer review. This synergy is poised to elevate the quality and clinical relevance of glioma radiomics, fostering a more robust and clinically viable body of research.