A reproducible statistical evaluation framework for large-sample assessment of AI-generated medical references: cross-platform application of the reference hallucination score Authors: Nurmuhammet Taş, Yakup Erden, Mustafa Hüseyin Temel, Fatih Bağcıer Published: 2026 ABSTRACT ObjectiveTo quantitatively evaluate the bibliographic reliability of AI-generated medical references across multiple chatbot platforms using the Reference Hallucination Score (RHS) and to examine the influence of output format on reference stability.MethodsIn this cross-sectional comparative study, three AI-based chatbots (ChatGPT, Gemini, and Perplexity) were prompted under 30 predefined medical subheadings in two output formats (letter and review article), generating 3,150 references. Reference reliability was assessed using the RHS, which evaluates presence/verifiability, bibliographic accuracy, PMID validity, and topic relevance. Due to non-normal distribution, data were analyzed using Kruskal–Wallis and Mann–Whitney U tests with Bonferroni correction. Effect sizes (η² and r) were calculated to distinguish statistical detectability from practical magnitude.ResultsSignificant inter-model differences were detected in total RHS scores (H(2) = 24.88, p < 0.001); however, the overall effect size was very small (η² = 0.007). Pairwise comparisons revealed statistically detectable differences between certain models, although effect sizes were consistently small (r = 0.02–0.20). Format-related differences were observed, with longer outputs demonstrating reduced bibliographic stability; however, these differences were numerically modest and partially attenuated after Bonferroni correction.ConclusionAI-based chatbots exhibit measurable bibliographic instability in reference generation, with statistically detectable but practically small differences between models. Beyond cross-platform comparison, this study proposes an instrument-agnostic statistical framework for large-sample evaluation of AI-generated reference reliability. Positioned within applied methodological research rather than AI benchmarking, the framework may serve as a template for future large-scale AI reliability studies. The proposed framework may contribute to methodological standardization in future evaluations of AI-generated scientific outputs.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1186/s12874-026-02906-0. Introduction Large language models (LLMs) and AI-based chatbots are increasingly integrated into academic writing and medical information workflows [1]. These systems are frequently used to generate literature summaries, draft manuscripts, and provide reference-supported responses to clinical queries. However, concerns have emerged regarding the reliability of AI-generated bibliographic references, particularly the generation of unverifiable, incomplete, or fabricated citations—commonly referred to as “reference hallucinations” [2]. Accurate referencing is a fundamental component of scientific integrity [3]. Bibliographic errors may compromise transparency, reproducibility, and the downstream validity of academic and clinical work. Several studies have documented the presence of hallucinated or fabricated references in AI-generated outputs, particularly in ChatGPT-generated medical texts [4–7]. Similar concerns have been raised across different medical domains, including nephrology and clinical education contexts [8, 9]. Fabricated DOIs and citation inaccuracies have also been systematically reported [10]. However, most investigations have focused on single-model evaluations or descriptive analyses, and standardized quantitative frameworks combined with robust statistical evaluation remain limited in cross-platform AI comparisons. Quantitative evaluation frameworks are essential for establishing transparent and reproducible standards in AI-assisted academic workflows. To date, no large-sample cross-platform evaluation has systematically integrated standardized scoring instruments, formal effect size estimation, and multiple-comparison control within a unified statistical framework. The Reference Hallucination Score (RHS), introduced by Aljamaan et al. [11], provides a structured method for assessing bibliographic reliability across predefined domains, including presence/verifiability, bibliographic accuracy, PMID validity, and topic relevance. Nevertheless, comparative evaluations using RHS across multiple chatbot systems under identical prompting conditions remain scarce. Moreover, few studies have incorporated effect size estimation and correction for multiple testing, which are essential for distinguishing statistical detectability from practical significance in large-sample AI evaluations. To address these methodological gaps, the present study conducted a cross-platform comparative analysis of three widely used AI-based chatbots (ChatGPT, Gemini, and Perplexity) using RHS as a quantitative evaluation framework. The selection of ChatGPT, Gemini, and Perplexity was based on their widespread public accessibility, high user adoption, and availability through web-based free-tier interfaces. These platforms represent commonly used tools in real-world academic and clinical workflows. In contrast, alternative systems such as Elicit or SciSpace are more specialized or less widely adopted by general end-users. Accordingly, the present study prioritized platforms that best reflect typical real-world user interaction scenarios, rather than specialized or domain-specific systems. A literature-rich and clinically common condition, plantar fasciitis [12–17], was selected as a model domain to enable structured and reproducible reference generation across standardized subtopics. The selection of plantar fasciitis was methodological rather than clinical; the condition was chosen due to its broad literature base and multidimensional subtopics, enabling robust cross-platform evaluation. In addition, the influence of output format (letter versus review article) on bibliographic stability was examined. By integrating non-parametric statistical testing, effect size reporting (η² and r), and Bonferroni correction for multiple comparisons, this study aims to provide a statistically rigorous evaluation of bibliographic reliability in AI-generated medical references and to contribute to the development of transparent methodological standards for assessing AI-assisted academic writing tools. Unlike prior descriptive evaluations, this study explicitly differentiates statistical significance from practical magnitude through formal effect size estimation, thereby reducing the risk of overinterpretation in large-sample AI assessments. This study positions itself within applied methodological research rather than AI benchmarking. Method Study design This study is a cross-sectional analysis planned to comparatively evaluate the accuracy and reliability of AI-based chatbots (ChatGPT, Gemini, and Perplexity) in literature-driven reference generation. The primary outcome measure was defined as the Reference Hallucination Score (RHS). The secondary objective was to examine the effect of different text formats (letter to the editor and review article) and pairwise comparisons between bots on reference accuracy. The inclusion of letter and review formats was based on the hypothesis that output length and structural complexity may influence the quality of reference generation. Review-style outputs typically require longer and denser reference lists, which may increase the likelihood of bibliographic inconsistencies compared to shorter, more constrained outputs such as letters. Example prompts for both formats are provided in the Supplementary Material (Additional file 1; see detailed scoring documentation). The 30 predefined subtopics were structured across six major clinical domains (pathophysiology, diagnosis, risk factors, complications, prevention, and rehabilitation) to ensure comprehensive and domain-balanced reference generation. Reference Hallucination Score (RHS) evaluation method In this study, the accuracy and reliability of references were scored using an adapted RHS. The evaluation was based on four main parameters: Presence/Verifiability (0–3 points): Availability of the reference in relevant databases (PubMed, Google Scholar).Bibliographic Accuracy (0–3 points): Correctness of journal name, year, volume, and page information.PMID Accuracy (0–2 points): Validity of the PubMed identification number and its correspondence to the correct publication.Relevance to Topic (0–3 points): Scientific relevance of the reference to the queried subject. Presence/Verifiability (0–3 points): Availability of the reference in relevant databases (PubMed, Google Scholar). Presence/Verifiability (0–3 points): Availability of the reference in relevant databases (PubMed, Google Scholar). Bibliographic Accuracy (0–3 points): Correctness of journal name, year, volume, and page information. Bibliographic Accuracy (0–3 points): Correctness of journal name, year, volume, and page information. PMID Accuracy (0–2 points): Validity of the PubMed identification number and its correspondence to the correct publication. PMID Accuracy (0–2 points): Validity of the PubMed identification number and its correspondence to the correct publication. Relevance to Topic (0–3 points): Scientific relevance of the reference to the queried subject. Relevance to Topic (0–3 points): Scientific relevance of the reference to the queried subject. Each reference was evaluated independently, and the sum of the obtained scores was recorded as the total RHS (range: 0–11). Higher RHS scores indicate fewer hallucinations and greater bibliographic reliability. All RHS evaluations were performed by a single investigator using a predefined scoring rubric operationalized a priori. Reference verification followed a standardized workflow in PubMed and Google Scholar to ensure consistent application of scoring criteria. Ambiguous cases were re-checked using the same decision rules before final scoring. To assess intra-rater scoring stability, a random subset of 200 references (approximately 6% of the dataset) was re-evaluated by the same investigator after a two-week interval using the identical rubric. Agreement between the first and second evaluations was 95%, indicating high intra-rater consistency. For overall comparisons, effect size for the Kruskal–Wallis test was calculated using eta-squared (η²), derived from the H statistic. The RHS rubric was applied using predefined decision rules to minimize subjective interpretation. All prompts, subheadings, and scoring criteria are available upon reasonable request to ensure methodological transparency. Detailed scoring rules, representative examples, and prompt templates for both formats are provided in the Supplementary Material (Additional file 1) to enhance transparency and reproducibility. Data collection Data were collected between 1 and 30 September 2025 using the publicly accessible web interfaces of ChatGPT (OpenAI), Gemini (Google), and Perplexity AI. All interactions were conducted using free-tier versions without paid subscription access, and no API access was employed. All prompts were originally entered in Turkish to reflect real-world user interaction. For reporting and standardization, prompts were translated into English in the manuscript and supplementary materials. At the time of data collection, publicly accessible web interfaces did not provide stable or transparently documented model version identifiers. Therefore, precise version tracking was not feasible. This limitation reflects authentic real-world end-user conditions rather than controlled developer-level API deployments. The study intentionally prioritized ecological validity and end-user reproducibility over version-controlled technical benchmarking, thereby aligning the evaluation with typical academic usage scenarios. The study focused on plantar fasciitis–related queries, and data were collected under a total of 30 predefined subheadings (Table 1). The full dataset of AI-generated references and corresponding RHS scores is publicly available at Zenodo (https://doi.org/10.5281/zenodo.19390459). Table 1Subtopics and Query Domains Used for Reference Hallucination Score (RHS) EvaluationMain TopicCategorySubtopics / Query ItemsPlantar FasciitisPathophysiology & Epidemiology• Pathophysiology and etiology of plantar fasciitis• Differences between acute and chronic plantar fasciitis• Incidence and prevalence• Clinical classification systemsDiagnosisClinical & Imaging Evaluation• Clinical signs and patient history• Physical examination tests (e.g., Windlass test, point tenderness)• Ultrasonography (US)• Magnetic resonance imaging (MRI)• Radiography and differential diagnosesRisk FactorsDemographic & Biomechanical• Common risk factors (obesity, prolonged standing, inappropriate footwear)• Biomechanical factors (pes planus, pes cavus, Achilles tendon tightness)• Effects of age and sex• Sports-related and occupational factors• Systemic diseases (e.g., diabetes mellitus, rheumatologic disorders)ComplicationsChronic Sequelae• Complications in chronic plantar fasciitis• Plantar fascia calcification and plantar spur formation• Chronic pain and functional limitations• Long-term consequences of untreated disease• Central sensitizationPreventionPreventive Strategies• Footwear selection• Occupational safety and ergonomic measures• Stretching and strengthening exercises for athletes• Strategies to reduce Achilles tendon tightness• Public education and awareness programsRehabilitationTreatment & Management• Effectiveness of conservative treatments (exercise therapy, orthoses, ESWT, dry needling)• Physical therapy modalities (ultrasound, laser, magnetic field therapy, shockwave therapy)• Indications for invasive treatments (injection therapies, PRP, prolotherapy, PNE)• Multidisciplinary rehabilitation approaches• Patient education and lifestyle modificationsESWT extracorporeal shock wave therapy, PRP platelet-rich plasma, PNE percutaneous needle electrolysisAll subtopics were entered verbatim into both letter and review prompt formats across platforms to ensure standardized and reproducible querying Table 1Subtopics and Query Domains Used for Reference Hallucination Score (RHS) EvaluationMain TopicCategorySubtopics / Query ItemsPlantar FasciitisPathophysiology & Epidemiology• Pathophysiology and etiology of plantar fasciitis• Differences between acute and chronic plantar fasciitis• Incidence and prevalence• Clinical classification systemsDiagnosisClinical & Imaging Evaluation• Clinical signs and patient history• Physical examination tests (e.g., Windlass test, point tenderness)• Ultrasonography (US)• Magnetic resonance imaging (MRI)• Radiography and differential diagnosesRisk FactorsDemographic & Biomechanical• Common risk factors (obesity, prolonged standing, inappropriate footwear)• Biomechanical factors (pes planus, pes cavus, Achilles tendon tightness)• Effects of age and sex• Sports-related and occupational factors• Systemic diseases (e.g., diabetes mellitus, rheumatologic disorders)ComplicationsChronic Sequelae• Complications in chronic plantar fasciitis• Plantar fascia calcification and plantar spur formation• Chronic pain and functional limitations• Long-term consequences of untreated disease• Central sensitizationPreventionPreventive Strategies• Footwear selection• Occupational safety and ergonomic measures• Stretching and strengthening exercises for athletes• Strategies to reduce Achilles tendon tightness• Public education and awareness programsRehabilitationTreatment & Management• Effectiveness of conservative treatments (exercise therapy, orthoses, ESWT, dry needling)• Physical therapy modalities (ultrasound, laser, magnetic field therapy, shockwave therapy)• Indications for invasive treatments (injection therapies, PRP, prolotherapy, PNE)• Multidisciplinary rehabilitation approaches• Patient education and lifestyle modificationsESWT extracorporeal shock wave therapy, PRP platelet-rich plasma, PNE percutaneous needle electrolysisAll subtopics were entered verbatim into both letter and review prompt formats across platforms to ensure standardized and reproducible querying Subtopics and Query Domains Used for Reference Hallucination Score (RHS) Evaluation • Pathophysiology and etiology of plantar fasciitis • Differences between acute and chronic plantar fasciitis • Incidence and prevalence • Clinical classification systems • Clinical signs and patient history • Physical examination tests (e.g., Windlass test, point tenderness) • Ultrasonography (US) • Magnetic resonance imaging (MRI) • Radiography and differential diagnoses • Common risk factors (obesity, prolonged standing, inappropriate footwear) • Biomechanical factors (pes planus, pes cavus, Achilles tendon tightness) • Effects of age and sex • Sports-related and occupational factors • Systemic diseases (e.g., diabetes mellitus, rheumatologic disorders) • Complications in chronic plantar fasciitis • Plantar fascia calcification and plantar spur formation • Chronic pain and functional limitations • Long-term consequences of untreated disease • Central sensitization • Footwear selection • Occupational safety and ergonomic measures • Stretching and strengthening exercises for athletes • Strategies to reduce Achilles tendon tightness • Public education and awareness programs • Effectiveness of conservative treatments (exercise therapy, orthoses, ESWT, dry needling) • Physical therapy modalities (ultrasound, laser, magnetic field therapy, shockwave therapy) • Indications for invasive treatments (injection therapies, PRP, prolotherapy, PNE) • Multidisciplinary rehabilitation approaches • Patient education and lifestyle modifications ESWT extracorporeal shock wave therapy, PRP platelet-rich plasma, PNE percutaneous needle electrolysis All subtopics were entered verbatim into both letter and review prompt formats across platforms to ensure standardized and reproducible querying For each subheading, requests were submitted in two different formats: Letter to the editor: 5 references requested.Review article: 30 references requested. Letter to the editor: 5 references requested. Letter to the editor: 5 references requested. Review article: 30 references requested. Review article: 30 references requested. Thus, 30 subheadings × 2 formats resulted in 60 outputs per chatbot. A total of 180 responses were evaluated across the three platforms. In terms of reference counts: Letter format: 30 × 5 = 150 references per chatbot.Review format: 30 × 30 = 900 references per chatbot. Letter format: 30 × 5 = 150 references per chatbot. Letter format: 30 × 5 = 150 references per chatbot. Review format: 30 × 30 = 900 references per chatbot. Review format: 30 × 30 = 900 references per chatbot. In total, 1,050 references per chatbot and 3,150 references overall were analyzed. The complete prompt templates, full subtopic list, and detailed RHS scoring rubric are provided in Additional file 2 to ensure full methodological reproducibility. Statistical analysis Data were analyzed using SPSS Statistics v.26 (IBM Corp., Armonk, NY, USA). As RHS scores did not meet the assumption of normality according to Kolmogorov–Smirnov and Shapiro–Wilk tests, non-parametric statistical tests were applied. Continuous variables are reported as mean ± standard deviation for descriptive clarity, while inferential comparisons were conducted using non-parametric methods due to distributional deviations. Comparisons among the three AI-based chatbots were performed using the Kruskal–Wallis H test. When significant differences were detected, post-hoc pairwise comparisons were conducted using the Mann–Whitney U test. To control for Type I error inflation due to multiple comparisons, Bonferroni correction was applied by adjusting the significance threshold (α/number of comparisons; adjusted α = 0.0167). Adjusted p-values are reported in the corresponding tables. Effect sizes for overall comparisons were estimated using eta-squared (η²) derived from the Kruskal–Wallis H statistic. For pairwise comparisons, effect sizes were calculated using the formula r = |Z|/√N, where N represents the combined sample size of the two compared groups. Effect sizes were interpreted according to Cohen’s conventional thresholds (0.10 = small, 0.30 = medium, 0.50 = large), acknowledging that these benchmarks are heuristic and context-dependent. Given the large sample size, effect size estimates were emphasized to distinguish statistical detectability from practical magnitude. A two-sided p-value < 0.05 was considered statistically significant unless otherwise adjusted for multiple comparisons. A comprehensive summary of all statistical analyses—including test statistics, raw and Bonferroni-adjusted p-values, and effect size estimates—is provided in the Supplementary Statistical Appendix (Additional file 3) and Summary of Statistical Analyses Performed in the Study (Additional file 4) to enhance transparency and reproducibility. The conceptual structure of the statistical evaluation framework implemented in the present study is illustrated in Fig. 1. Fig. 1Reproducible Statistical Evaluation Framework for Large-Sample Assessment of AI-Generated References Reproducible Statistical Evaluation Framework for Large-Sample Assessment of AI-Generated References Ethical considerations The study did not involve human or animal participants and was conducted solely on publicly available AI chatbot outputs. Therefore, ethical committee approval was not required. Results Descriptive statistics are presented as medians with interquartile ranges (IQR) to ensure consistency with non-parametric analyses. RHS scores were consistently high for ChatGPT and Perplexity (median 11, IQR 11–11), whereas Gemini demonstrated greater variability (median 9, IQR 7–11), indicating broader dispersion in reference reliability. For completeness and ease of comparison with prior literature, mean ± standard deviation values are also reported in Tables 2, 3, 4, 5 and 6. As RHS scores did not meet normality assumptions, non-parametric tests were used for all inferential comparisons. Total RHS scores and subcomponents are summarized in Tables 2, 3, 4, 5 and 6. Table 2Reference Hallucination Score ComparisonCriterionChatGPTGeminiPerplexity p Presence / Verifiability (0–3)2.95 ± 0.34 (0–3)2.87 ± 0.56 (0–3)2.80 ± 0.66 (0–3)< 0.001Bibliographic Accuracy (0–3)2.72 ± 0.67 (0–3)2.82 ± 0.61 (0–3)2.70 ± 0.73 (0–3)< 0.001PMID Accuracy (0–2)1.85 ± 0.48 (0–2)1.84 ± 0.54 (0–2)1.72 ± 0.64 (0–2)< 0.001Relevance to Topic (0–3)2.88 ± 0.38 (0–3)2.85 ± 0.61 (0–3)2.88 ± 0.41 (0–3)0.040 TOTAL RHS (0–11) 10.39 ± 1.35 (0–11) 10.38 ± 1.64 (0–11) 10.11 ± 2.08 (0–11) < 0.001 Values are presented as mean ± standard deviation (range) Table 2Reference Hallucination Score ComparisonCriterionChatGPTGeminiPerplexity p Presence / Verifiability (0–3)2.95 ± 0.34 (0–3)2.87 ± 0.56 (0–3)2.80 ± 0.66 (0–3)< 0.001Bibliographic Accuracy (0–3)2.72 ± 0.67 (0–3)2.82 ± 0.61 (0–3)2.70 ± 0.73 (0–3)< 0.001PMID Accuracy (0–2)1.85 ± 0.48 (0–2)1.84 ± 0.54 (0–2)1.72 ± 0.64 (0–2)< 0.001Relevance to Topic (0–3)2.88 ± 0.38 (0–3)2.85 ± 0.61 (0–3)2.88 ± 0.41 (0–3)0.040 TOTAL RHS (0–11) 10.39 ± 1.35 (0–11) 10.38 ± 1.64 (0–11) 10.11 ± 2.08 (0–11) < 0.001 Values are presented as mean ± standard deviation (range) Reference Hallucination Score Comparison Values are presented as mean ± standard deviation (range) Table 3RHS Comparison Between Bots for Letter Format (Excel-Based, Kruskal–Wallis)CriterionChatGPTGeminiPerplexityp-valuePresence / Verifiability (0–3)3.00 ± 0.002.91 ± 0.482.89 ± 0.450.006Bibliographic Accuracy (0–3)2.94 ± 0.262.88 ± 0.522.79 ± 0.620.012PMID Accuracy (0–2)1.97 ± 0.201.91 ± 0.421.67 ± 0.68< 0.001Relevance to Topic (0–3)2.93 ± 0.292.91 ± 0.442.93 ± 0.390.713 TOTAL RHS Score (0–11) 10.84 ± 0.48 10.61 ± 1.40 10.28 ± 1.75 < 0.001 RHS Reference Hallucination Score Table 3RHS Comparison Between Bots for Letter Format (Excel-Based, Kruskal–Wallis)CriterionChatGPTGeminiPerplexityp-valuePresence / Verifiability (0–3)3.00 ± 0.002.91 ± 0.482.89 ± 0.450.006Bibliographic Accuracy (0–3)2.94 ± 0.262.88 ± 0.522.79 ± 0.620.012PMID Accuracy (0–2)1.97 ± 0.201.91 ± 0.421.67 ± 0.68< 0.001Relevance to Topic (0–3)2.93 ± 0.292.91 ± 0.442.93 ± 0.390.713 TOTAL RHS Score (0–11) 10.84 ± 0.48 10.61 ± 1.40 10.28 ± 1.75 < 0.001 RHS Reference Hallucination Score RHS Comparison Between Bots for Letter Format (Excel-Based, Kruskal–Wallis) RHS Reference Hallucination Score Table 4RHS Score Comparison in Article Format (Excel-Based Recalculation)CriterionChatGPT (mean ± SD, min–max)Gemini (mean ± SD, min–max)Perplexity (mean ± SD, min–max)p-valuePresence / Verifiability (0–3)2.93 ± 0.36 (0–3)2.86 ± 0.58 (0–3)2.78 ± 0.69 (0–3)< 0.001Bibliographic Accuracy (0–3)2.68 ± 0.70 (0–3)2.81 ± 0.63 (0–3)2.69 ± 0.75 (0–3)< 0.001PMID Accuracy (0–2)1.83 ± 0.51 (0–2)1.83 ± 0.55 (0–2)1.73 ± 0.63 (0–2)< 0.001Relevance to Topic (0–3)2.89 ± 0.78 (0–3)2.84 ± 0.63 (0–3)2.88 ± 0.41 (0–3)0.050 TOTAL RHS (0–11) 10.34 ± 1.58 (0–11) 10.35 ± 1.67 (2–11) 10.08 ± 2.13 (0–11) < 0.001 RHS Reference Hallucination ScorePairwise post-hoc comparisons with Bonferroni adjustment are reported in the textp-values represent Kruskal–Wallis overall comparisons Table 4RHS Score Comparison in Article Format (Excel-Based Recalculation)CriterionChatGPT (mean ± SD, min–max)Gemini (mean ± SD, min–max)Perplexity (mean ± SD, min–max)p-valuePresence / Verifiability (0–3)2.93 ± 0.36 (0–3)2.86 ± 0.58 (0–3)2.78 ± 0.69 (0–3)< 0.001Bibliographic Accuracy (0–3)2.68 ± 0.70 (0–3)2.81 ± 0.63 (0–3)2.69 ± 0.75 (0–3)< 0.001PMID Accuracy (0–2)1.83 ± 0.51 (0–2)1.83 ± 0.55 (0–2)1.73 ± 0.63 (0–2)< 0.001Relevance to Topic (0–3)2.89 ± 0.78 (0–3)2.84 ± 0.63 (0–3)2.88 ± 0.41 (0–3)0.050 TOTAL RHS (0–11) 10.34 ± 1.58 (0–11) 10.35 ± 1.67 (2–11) 10.08 ± 2.13 (0–11) < 0.001 RHS Reference Hallucination ScorePairwise post-hoc comparisons with Bonferroni adjustment are reported in the textp-values represent Kruskal–Wallis overall comparisons RHS Score Comparison in Article Format (Excel-Based Recalculation) RHS Reference Hallucination Score Pairwise post-hoc comparisons with Bonferroni adjustment are reported in the text p-values represent Kruskal–Wallis overall comparisons Table 5Comparison of Letter vs. Article Formats Based on RHS Scores (Mann–Whitney U Test)ChatGPT: Article vs. LetterCriterionp-valueInterpretationPresence / Verifiability0.0036Article format demonstrated significantly lower RHS scoresBibliographic Accuracy0.2278No significant differencePMID Accuracy0.0076Article format significantly lower RHS scoresRelevance to Topic0.2122No significant difference TOTAL RHS 0.0015 Article format significantly lower RHS scores Perplexity: Article vs. LetterCriterionp-valueInterpretationPresence / Verifiability0.1770No significant differenceBibliographic Accuracy0.1770No significant differencePMID Accuracy0.1770No significant differenceRelevance to Topic0.1770No significant difference TOTAL RHS 0.1770 No significant difference Gemini: Article vs. LetterCriterionp-valueInterpretationPresence / Verifiability0.0750No significant differenceBibliographic Accuracy0.2700No significant differencePMID Accuracy0.0560Borderline, not statistically significantRelevance to Topic0.1210No significant difference TOTAL RHS 0.0196 Not statistically significant after Bonferroni adjustment RHS Reference Hallucination Score Table 5Comparison of Letter vs. Article Formats Based on RHS Scores (Mann–Whitney U Test)ChatGPT: Article vs. LetterCriterionp-valueInterpretationPresence / Verifiability0.0036Article format demonstrated significantly lower RHS scoresBibliographic Accuracy0.2278No significant differencePMID Accuracy0.0076Article format significantly lower RHS scoresRelevance to Topic0.2122No significant difference TOTAL RHS 0.0015 Article format significantly lower RHS scores Perplexity: Article vs. LetterCriterionp-valueInterpretationPresence / Verifiability0.1770No significant differenceBibliographic Accuracy0.1770No significant differencePMID Accuracy0.1770No significant differenceRelevance to Topic0.1770No significant difference TOTAL RHS 0.1770 No significant difference Gemini: Article vs. LetterCriterionp-valueInterpretationPresence / Verifiability0.0750No significant differenceBibliographic Accuracy0.2700No significant differencePMID Accuracy0.0560Borderline, not statistically significantRelevance to Topic0.1210No significant difference TOTAL RHS 0.0196 Not statistically significant after Bonferroni adjustment RHS Reference Hallucination Score Comparison of Letter vs. Article Formats Based on RHS Scores (Mann–Whitney U Test) RHS Reference Hallucination Score Table 6p-Values for Pairwise Bot Comparisons by FormatFormat / CriterionChatGPT vs. GeminiChatGPT vs. PerplexityGemini vs. PerplexityLetter / Presence/Verifiability0.0030.2400.004Letter / Bibliographic Accuracy1.0001.0001.000Letter / PMID Accuracy0.0280.0020.582Letter / Relevance to Topic0.3300.4650.041Letter / TOTAL RHS 0.057 0.015 0.001 Article / Presence/Verifiability0.003< 0.0010.097Article / Bibliographic Accuracy0.7261.0000.726Article / PMID Accuracy0.5820.5821.000Article / Relevance to Topic1.0000.1490.149Article / TOTAL RHS 0.058 0.001 0.047 RHS: Reference Hallucination ScoreValues below this threshold are considered statistically significantpairwise comparisons were evaluated using Bonferroni-adjusted significance thresholds (α = 0.0167) Table 6p-Values for Pairwise Bot Comparisons by FormatFormat / CriterionChatGPT vs. GeminiChatGPT vs. PerplexityGemini vs. PerplexityLetter / Presence/Verifiability0.0030.2400.004Letter / Bibliographic Accuracy1.0001.0001.000Letter / PMID Accuracy0.0280.0020.582Letter / Relevance to Topic0.3300.4650.041Letter / TOTAL RHS 0.057 0.015 0.001 Article / Presence/Verifiability0.003< 0.0010.097Article / Bibliographic Accuracy0.7261.0000.726Article / PMID Accuracy0.5820.5821.000Article / Relevance to Topic1.0000.1490.149Article / TOTAL RHS 0.058 0.001 0.047 RHS: Reference Hallucination ScoreValues below this threshold are considered statistically significantpairwise comparisons were evaluated using Bonferroni-adjusted significance thresholds (α = 0.0167) p-Values for Pairwise Bot Comparisons by Format RHS: Reference Hallucination Score Values below this threshold are considered statistically significant pairwise comparisons were evaluated using Bonferroni-adjusted significance thresholds (α = 0.0167) To enhance interpretability, proportions of PMID accuracy categories were calculated. ChatGPT demonstrated 89.8% correct, 5.2% mismatched, and 5.0% invalid references. Gemini showed the highest proportion of correct PMIDs (92.1%), with minimal mismatches (0.2%) and 7.7% invalid entries. Perplexity exhibited lower accuracy, with 83.0% correct, 6.4% mismatched, and 10.6% invalid references. Detailed distributions are provided in Supplementary Material (Additional file 5). These proportion-based findings provide a more interpretable complement to mean-based RHS comparisons. When all formats were analyzed together, a statistically significant difference in total RHS scores was observed among the three chatbots (Kruskal–Wallis H [2] = 24.88, p < 0.001). The overall effect size was very small (η² = 0.007), indicating that model differences accounted for less than 1% of the variance in total RHS scores. Pairwise comparisons showed that ChatGPT scored higher than Gemini (U = 499,769.5, Z = − 4.89, p < 0.001, r = 0.11) and Gemini scored higher than Perplexity (U = 588,454.5, Z = − 3.66, p < 0.001, r = 0.08), whereas no significant difference was detected between ChatGPT and Perplexity (p = 0.265). Although statistically significant, effect sizes were small, indicating limited practical differentiation between models. Differences in Presence/Verifiability, Bibliographic Accuracy, and PMID Accuracy were statistically significant (all p < 0.001), whereas Relevance to Topic demonstrated only marginal variation (p = 0.040) and did not remain statistically significant after Bonferroni correction (adjusted α = 0.0167). In letter format, total RHS scores differed significantly among models (H [2] = 15.19, p < 0.001). ChatGPT performed significantly better than Perplexity (U = 13,020, Z = − 3.44, p < 0.001, r = 0.20), and Gemini also outperformed Perplexity (U = 12,793.5, Z = − 2.96, p = 0.003, r = 0.17). No significant difference was observed between ChatGPT and Gemini (p = 0.79). Inter-model differences were primarily driven by the PMID Accuracy component. No significant variation was observed in Relevance to Topic (p = 0.713). A ceiling effect was noted in ChatGPT’s Presence/Verifiability scores. In article format, a significant overall difference was detected (H [2] = 26.78, p < 0.001). ChatGPT demonstrated higher total RHS scores than Gemini (U = 360,428, Z = − 5.18, p < 0.001, r = 0.12). Differences between ChatGPT and Perplexity (U = 384,159, Z = − 2.34, p = 0.019, r = 0.06) and between Gemini and Perplexity (U = 427,877.5, Z = − 2.82, p = 0.005, r = 0.07) were statistically detectable; however, all effect sizes were small. The Relevance to Topic criterion yielded a borderline result (p = 0.050) and did not demonstrate clear statistical significance. Bonferroni correction was applied to all post-hoc pairwise comparisons (adjusted α = 0.0167). After adjustment, overall differences between ChatGPT and Gemini and between Gemini and Perplexity remained statistically significant, whereas the difference between ChatGPT and Perplexity did not. In letter format, comparisons involving Perplexity remained significant after correction. In article format, the difference between ChatGPT and Gemini and between Gemini and Perplexity remained significant; however, the ChatGPT–Perplexity difference did not remain statistically significant following adjustment (adjusted p = 0.058). In cross-format analysis, ChatGPT demonstrated higher RHS scores in article format compared to letter format, particularly in Presence/Verifiability (p = 0.0036), PMID Accuracy (p = 0.0076), and Total RHS (p = 0.0015). For Gemini, only total RHS differed significantly between formats (p = 0.0196). No significant format-related differences were observed for Perplexity. Overall, while multiple statistically detectable differences were identified, effect sizes were consistently small, suggesting that absolute performance differences between models were numerically modest despite statistical detectability. Discussion In this study, the reference reliability of three AI-based chatbots (ChatGPT, Gemini, and Perplexity) was comparatively evaluated within the context of plantar fasciitis literature using the Reference Hallucination Score (RHS). Although statistically significant differences were observed, effect sizes were consistently small, and no system demonstrated error-free reference generation. Our findings are broadly consistent with previous research reporting that AI-based chatbots may generate unverifiable or fabricated references in medical contexts [4–10]. The phenomenon of reference hallucination has been widely discussed in the context of natural language generation systems [2], and concerns regarding bibliographic inaccuracies have been raised across multiple clinical domains [6–10]. The RHS methodology introduced by Aljamaan et al. [11] enabled structured quantitative comparison of bibliographic reliability. However, most prior studies have focused on single-model evaluations. By comparatively examining multiple chatbots under identical methodological conditions, the present study extends the literature by providing cross-platform data within a clinically relevant and literature-rich domain. An additional contribution of this study is the evaluation of format-related effects on reference reliability. Differences between letter and article formats suggest that longer and reference-dense outputs may be associated with reduced bibliographic stability. However, the observed deterioration remained numerically limited. These findings imply that reference hallucinations may be influenced by output length and structural complexity, but the extent of this effect should be interpreted cautiously. Importantly, although several comparisons reached statistical significance, corresponding effect sizes were consistently small (η² < 0.01; r values between 0.02 and 0.20). The very small overall η² indicates that model type explained less than 1% of the variance in RHS scores. The large sample size likely contributed to statistical detectability despite minimal absolute score differences. Furthermore, some pairwise comparisons—particularly between ChatGPT and Perplexity in article format—did not remain statistically significant after Bonferroni correction. Collectively, these findings support the interpretation that inter-model differences, while statistically observable, were numerically modest and of limited practical magnitude. To enhance interpretability, effect size metrics were complemented by proportion-based summaries of reference accuracy (correct, mismatched, and invalid references), providing a more practical and clinically meaningful understanding of differences between models. Proportion-based analyses further demonstrated that these differences correspond to relatively small absolute changes in reference accuracy. Although all models exhibited high proportions of correct references, non-negligible rates of invalid and mismatched citations persisted across platforms, indicating that effect sizes (r ≈ 0.1) reflect modest practical differences rather than clear superiority of any single system. These findings have direct implications for real-world academic and clinical use. From a clinical and academic perspective, even small rates of bibliographic inaccuracy may carry implications. Incorrect or unverifiable references can undermine scientific transparency [3] and potentially influence downstream research synthesis. Therefore, regardless of model choice, AI-generated references should not be used without independent verification through established databases such as PubMed or Google Scholar. AI-based systems may serve as supportive drafting tools in academic workflows, but they should not be considered validated bibliographic authorities. The consistent discrepancy between statistical detectability and practical magnitude underscores the importance of reporting effect sizes in large-sample AI evaluations and complementing them with interpretable proportion-based metrics. The present findings not only apply RHS to a cross-platform context but also provide indirect empirical support for its large-sample stability and cross-format robustness. The consistency of effect size patterns across output formats suggests that RHS maintains structural stability under heterogeneous prompting conditions. Beyond cross-platform comparison, this study contributes methodologically by demonstrating the scalability, statistical robustness, and reproducibility of the RHS framework in large-sample AI evaluations. Collectively, these elements define a replicable statistical evaluation architecture for large-scale assessment of AI-generated references. Building upon this foundation, we propose a structured statistical framework to guide future studies evaluating bibliographic reliability in AI-generated outputs. The conceptual structure of the proposed framework is illustrated in Fig. 1. Use of a structured and operationalized scoring instrument with clearly defined domains and reproducible decision rules (e.g., RHS). Assessment of distributional assumptions prior to inferential testing, followed by appropriate selection of parametric or non-parametric methods. This step is intended as a methodological safeguard rather than a characterization of model performance, and direct use of non-parametric methods may be appropriate in high-accuracy contexts.Mandatory reporting of effect sizes alongside p-values to distinguish statistical detectability from practical magnitude, particularly in large datasets.Application of multiple-comparison correction procedures to control Type I error inflation in post-hoc analyses.Provision of transparent supplementary statistical documentation, including formula derivations and scoring stability procedures, to enhance reproducibility. Assessment of distributional assumptions prior to inferential testing, followed by appropriate selection of parametric or non-parametric methods. This step is intended as a methodological safeguard rather than a characterization of model performance, and direct use of non-parametric methods may be appropriate in high-accuracy contexts. Assessment of distributional assumptions prior to inferential testing, followed by appropriate selection of parametric or non-parametric methods. This step is intended as a methodological safeguard rather than a characterization of model performance, and direct use of non-parametric methods may be appropriate in high-accuracy contexts. Mandatory reporting of effect sizes alongside p-values to distinguish statistical detectability from practical magnitude, particularly in large datasets. Mandatory reporting of effect sizes alongside p-values to distinguish statistical detectability from practical magnitude, particularly in large datasets. Application of multiple-comparison correction procedures to control Type I error inflation in post-hoc analyses. Application of multiple-comparison correction procedures to control Type I error inflation in post-hoc analyses. Provision of transparent supplementary statistical documentation, including formula derivations and scoring stability procedures, to enhance reproducibility. Provision of transparent supplementary statistical documentation, including formula derivations and scoring stability procedures, to enhance reproducibility. This framework is designed to minimize overinterpretation, enhance statistical rigor, and promote reproducible methodological standards for evaluating AI-generated scientific content across heterogeneous platforms and output formats. Although operationalized using the RHS in the present study, the framework is instrument-agnostic and may be generalized to other structured evaluation systems developed to assess the reliability and validity of AI-generated scientific outputs. It is important to emphasize that RHS evaluates bibliographic reliability rather than semantic accuracy, clinical validity, or scientific coherence. Therefore, the findings should not be interpreted as a comprehensive assessment of overall AI performance, but rather as an evaluation of bibliographic stability within AI-generated reference lists. Several limitations should be acknowledged when interpreting these findings. First, the study evaluated only three AI-based chatbots; inclusion of additional contemporary systems would enhance generalizability. Second, the analysis was restricted to plantar fasciitis literature [12–17], which may limit generalizability across medical domains. However, this restriction was an intentional methodological design choice. Plantar fasciitis was selected as a literature-rich, well-indexed, and multidimensional condition to enable controlled and standardized cross-platform comparisons under identical prompting conditions. Restricting the domain minimized confounding variability related to topic complexity, terminology, and evidence distribution across different clinical fields. The primary focus was methodological standardization rather than clinical scope generalization. Nevertheless, future studies incorporating multiple medical domains are warranted to validate the proposed framework across heterogeneous contexts. Third, RHS scoring was conducted by a single evaluator, which may introduce subjective bias. However, several measures were implemented to mitigate this limitation. A strictly predefined and operationalized scoring rubric was applied, and a standardized verification workflow using PubMed and Google Scholar was followed for all references. In addition, intra-rater reliability was high (95% agreement), supporting scoring consistency. To further enhance transparency, detailed scoring rules and representative examples are provided in the Supplementary Material (Additional file 1), enabling independent verification and reproducibility. Nevertheless, future studies incorporating multi-rater validation are warranted to strengthen generalizability. While RHS was originally developed and validated by Aljamaan et al., the present study did not aim to conduct formal psychometric re-validation (e.g., factor structure analysis or inter-rater reliability modeling). Rather, the primary objective was comparative platform evaluation using an established instrument under standardized conditions. Future studies integrating reference accuracy with semantic content validation and clinical appropriateness measures would provide a more comprehensive evaluation of AI-assisted academic writing tools. Conclusion This study evaluated the bibliographic reliability of AI-generated references using the Reference Hallucination Score (RHS). Although statistically significant differences were observed among models, effect sizes were consistently small, indicating limited practical differentiation. Importantly, RHS reflects bibliographic stability rather than overall content accuracy. The findings indicate that AI-based chatbots may exhibit instability in reference generation and therefore require independent manual verification before use in academic or clinical contexts. AI systems should be regarded as supportive drafting tools rather than validated bibliographic authorities. Beyond model comparison, this study provides a reproducible methodological framework integrating effect size estimation, multiple-comparison control, and transparent reporting standards. This framework may inform future large-scale evaluations of AI-generated scientific content and contribute to methodological standardization in this emerging field. Supplementary Information Supplementary Material 1. Prompt templates and representative examples for letter and review formats used in RHS evaluation. Supplementary Material 1. Prompt templates and representative examples for letter and review formats used in RHS evaluation. Supplementary Material 2. Full prompt templates, subtopic list, and RHS scoring rubric Supplementary Material 2. Full prompt templates, subtopic list, and RHS scoring rubric Supplementary Material 3. Statistical Appendix (η², r, Bonferroni, intra-rater formulas) Supplementary Material 3. Statistical Appendix (η², r, Bonferroni, intra-rater formulas) Supplementary Material 4. Summary of Statistical Analyses Performed in the Study Supplementary Material 4. Summary of Statistical Analyses Performed in the Study Supplementary Material 5. Distribution of PMID accuracy categories across chatbot platforms (n and %) Supplementary Material 5. Distribution of PMID accuracy categories across chatbot platforms (n and %)