Performance of large language models in urological decision support: a guideline-based comparative evaluation in urolithiasis Authors: Adem Tunçekin, Yasin Aktaş Published: 2026 ABSTRACT Large language models (LLMs) are increasingly investigated for their potential role in guideline-based clinical information support. However, their consistency with subspecialty guidelines, particularly in urolithiasis, remains underexplored. This study aimed to evaluate the performance of four large language models (LLMs); GPT-4, GPT-4-turbo, Claude, and Gemini in generating guideline-concordant responses to clinical questions related to urolithiasis. A total of 105 clinical questions were independently developed by the authors based on urolithiasis management principles. Each LLM generated responses in two separate sessions. Two experienced urologists evaluated the outputs for accuracy and concordance with guideline recommendations. Inter-rater agreement analysis demonstrated fair agreement between evaluators. Differences across models and guideline categories were assessed using appropriate statistical tests. All four LLMs demonstrated high guideline adherence, with mean total scores ranging from 86.5 ± 5.2 (Gemini) to 92.8 ± 3.1 (Claude). Claude achieved the highest correlation with expert ratings (r = 0.94, p < 0.01). There were no statistically significant differences across models or among the nine clinical categories (p > 0.05). Session-to-session repeatability was also high for all models, with intra-model correlation coefficients exceeding 0.90. LLMs, particularly Claude, can provide reliable, guideline-consistent answers to urolithiasis-related clinical queries. Their consistent performance across themes suggests utility as adjunctive informational tools for guideline-based urological education and support, although further validation in real-world clinical settings remains necessary. Introduction Large language models (LLMs), including platforms such as GPT-4, GPT-4-turbo, Claude and Gemini, have rapidly emerged as tools with potential applications in healthcare decision-making. These models, trained on vast textual corpora, are capable of generating human-like responses to complex clinical prompts. As artificial intelligence (AI) continues to integrate into clinical workflows, evaluating the reliability and medical accuracy of these models has become increasingly critical [1, 2]. In particular, the role of LLMs in supporting adherence to established clinical practice guidelines has gained attention. Ensuring that AI-generated medical content aligns with evidence-based recommendations is essential to prevent misinformation and enhance clinical safety. Several recent studies have evaluated LLM performance across various medical domains, including oncology, internal medicine, and chronic disease management [3–5]. Moreover, early work within urology has assessed ChatGPT’s performance in testicular cancer and urolithiasis contexts [6–8]. However, comparative analyses across multiple LLM platforms and their agreement with expert assessments remain limited [9]. This study aimed to assess and compare the compliance of four widely used LLMs; GPT-4, GPT-4-turbo, Claude and Gemini with the European Association of Urology (EAU) guidelines for urolithiasis. By employing expert scoring and consensus methodology, we sought to determine the degree of alignment between AI-generated answers and evidence-based clinical standards. Methods Study design This was a cross-sectional, comparative evaluation study designed to assess the compliance of large language models (LLMs) with the European Association of Urology (EAU) guidelines for urolithiasis. A total of 105 standardized clinical questions were developed by the authors, reflecting key recommendations based on EAU guidance. These questions were independently submitted to four state-of-the-art LLMs; GPT-4, GPT-4-turbo, Claude and Gemini. The responses generated by each model were subsequently evaluated by two urologists with expertise in guideline-based clinical decision-making. Each response was scored on a 3-point ordinal scale: 0 = non-compliant, 1 = partially compliant, 2 = fully compliant, based on its alignment with the EAU guideline recommendations. In instances where the two urologists provided differing scores, a consensus was reached through joint discussion. This study did not involve any human subjects or identifiable patient data. Therefore, approval from an institutional ethics committee was not required. Large language models and question framework The evaluation included four LLMs: GPT-4 and GPT-4-turbo (developed by OpenAI), Gemini (developed by Google DeepMind), and Claude (developed by Anthropic). All models were accessed between June 2025-September 2025 via their publicly available web interfaces under default settings. No fine-tuning or prompt engineering beyond the base query was applied. Each model generated responses during two separate sessions using identical question sets and standardized prompting procedures. The sessions were conducted independently under the same usage conditions in order to minimize contextual variability and evaluate response consistency across repeated interactions. To ensure methodological consistency, all clinical questions were submitted using identical wording across all large language models. Each query was entered in a newly initiated chat session without follow-up prompts or conversational memory carryover. No additional contextual guidance, prompt engineering, or iterative clarification was provided. All models were evaluated using their publicly accessible web interfaces under default settings during the same study period. The question set was developed by two urologists with clinical experience in endourology and guideline-based management of urolithiasis and categorized into 9 predefined thematic parts, each representing a distinct clinical domain relevant to urolithiasis. These parts included: [1] evaluation and diagnosis [2], acute management [3], ureteric stones [4], renal stones [5], pediatric stone disease [6], stone disease in pregnancy [7], infectious stones [8], metabolic evaluation, and [9] follow-up and recurrence. The categorization was not based on formal EAU guideline sections but was structured by the research team to reflect clinically meaningful groupings. Questions were derived from key diagnostic and therapeutic recommendations within the EAU urolithiasis guidelines, and efforts were made to ensure balanced representation of major guideline topics and varying levels of clinical complexity during question development. The full list of 105 clinical questions, categorized by urolithiasis-related themes, is provided in Supplementary Appendix A. These questions were used consistently across all evaluated LLMs. Although formal external validation or pilot testing was not performed, the study was designed as an exploratory comparative evaluation of guideline concordance rather than a formal psychometric validation study. Statistical analysis All statistical analyses were conducted using IBM SPSS Statistics for Windows, Version 23.0 (IBM Corp., Armonk, NY, USA). Given the ordinal nature of the expert consensus scores, non-parametric tests were employed for all inferential procedures. The Friedman test was applied to assess overall performance differences among the four large language models across the full set of 105 questions. To investigate model performance variations across different thematic parts of the guideline-based question set, separate Kruskal–Wallis H tests were performed for each predefined category (n = 9). Spearman’s rank-order correlation coefficients were calculated to evaluate the degree of alignment between individual model scores and the final expert consensus. A two-tailed p-value < 0.05 was considered statistically significant in all analyses. Inter-rater agreement between the two independent expert evaluators was assessed using Cohen’s kappa coefficient prior to consensus resolution. Results Overall distribution of consensus scores across models Descriptive statistics demonstrated that all four large language models achieved high levels of compliance with the EAU urolithiasis guidelines, with median consensus scores approaching 2.0. GPT-4-turbo recorded the highest mean consensus score (1.87 ± 0.34), while GPT-4 showed the lowest (1.81 ± 0.40), although overall variability among models was minimal. The Friedman test revealed no statistically significant difference in consensus scores among the models (χ² = 0.99, df = 3, p = 0.804), indicating a comparable level of performance in guideline adherence across GPT-4, GPT-4-turbo, Claude, and Gemini (Table 1). Table 1Descriptive statistics and overall comparison of question-level consensus scores between language modelsModelMean ± SDMedian (IQR)Min–MaxFriedman χ²p valueGPT-41.81 ± 0.402.0 (1–2)1–20.990.804GPT-4-turbo1.87 ± 0.342.0 (2–2)1–2Gemini1.82 ± 0.382.0 (1–2)1–2Claude1.86 ± 0.362.0 (2–2)1–2SD: Standard deviation; IQR: Interquartile range; GPT: Generative Pretrained Transformer Table 1Descriptive statistics and overall comparison of question-level consensus scores between language modelsModelMean ± SDMedian (IQR)Min–MaxFriedman χ²p valueGPT-41.81 ± 0.402.0 (1–2)1–20.990.804GPT-4-turbo1.87 ± 0.342.0 (2–2)1–2Gemini1.82 ± 0.382.0 (1–2)1–2Claude1.86 ± 0.362.0 (2–2)1–2SD: Standard deviation; IQR: Interquartile range; GPT: Generative Pretrained Transformer Descriptive statistics and overall comparison of question-level consensus scores between language models SD: Standard deviation; IQR: Interquartile range; GPT: Generative Pretrained Transformer There were no statistically significant differences in consensus scores across the four large language models based on the Friedman test (p = 0.804). Subgroup analysis by question categories To explore whether the compliance of large language models (LLMs) varied across different domains of the EAU urolithiasis guidelines, a subgroup analysis was conducted based on predefined question categories (n = 9 parts). The Kruskal–Wallis H test was applied to compare consensus scores among the four LLMs; GPT-4, GPT-4-turbo, Claude, and Gemini within each question category. While minor variations in median scores were observed, none of the comparisons reached statistical significance (p > 0.05), indicating no significant differences in performance across the guideline domains (Table 2). Table 2Comparison of consensus scores among language models across question categoriesQuestion Partsχ² (df = 3)p value1. Evaluation and Diagnosis1.4060.7042. Acute Management0.9340.8173. Ureteric Stones2.4740.4804. Renal Stones1.2400.7435. Pediatric Stone Disease1.2300.7456. Stone Disease in Pregnancy1.0660.7857. Infectious Stones0.8050.8498. Metabolic Evaluation1.3370.7209. Follow-up and Recurrence1.7810.620 Table 2Comparison of consensus scores among language models across question categoriesQuestion Partsχ² (df = 3)p value1. Evaluation and Diagnosis1.4060.7042. Acute Management0.9340.8173. Ureteric Stones2.4740.4804. Renal Stones1.2400.7435. Pediatric Stone Disease1.2300.7456. Stone Disease in Pregnancy1.0660.7857. Infectious Stones0.8050.8498. Metabolic Evaluation1.3370.7209. Follow-up and Recurrence1.7810.620 Comparison of consensus scores among language models across question categories All Kruskal–Wallis tests were non-significant (p > 0.05), indicating no statistically meaningful variation in model performance across the question categories. Correlation between large language model consensus scores and expert agreement Inter-rater agreement analysis demonstrated fair agreement between the two evaluators across all evaluated models prior to consensus resolution (Cohen’s κ range: 0.284–0.324, all p < 0.001). The analysis revealed strong and statistically significant positive correlations between the consensus scores generated by each large language model (LLM) and the expert-derived consensus ratings, reflecting a high degree of alignment. Among the models, Claude exhibited the strongest correlation with expert consensus (ρ = 0.964, p < 0.001), followed closely by GPT-4-turbo (ρ = 0.943, p < 0.001), GPT-4 (ρ = 0.938, p < 0.001), and Gemini (ρ = 0.862, p < 0.001). All correlations were statistically significant, indicating robust associations between each model’s outputs and expert judgment. Claude demonstrated the highest alignment, suggesting superior consistency with expert-level decision-making. Discussion This study demonstrates that current large language models (LLMs) GPT-4, GPT-4-turbo, Claude, and Gemini are capable of generating responses with high adherence to the European Association of Urology (EAU) guidelines for the diagnosis and management of urolithiasis [10]. All four models achieved median consensus scores near full compliance, with no statistically significant differences among them. These findings highlight a key advancement in the ability of general-purpose LLMs to apply specialty-specific recommendations without explicit fine-tuning. Unlike traditional rule-based decision-support systems, LLMs rely on extensive pretraining across diverse data sources to generate clinically aligned content, offering potential utility for evidence-based informational support in clinical environments [11]. The present study should therefore be interpreted as an exploratory evaluation of guideline concordance rather than a direct assessment of autonomous clinical decision-making capability. Our findings are consistent with previous studies that examined the performance of AI models in the context of urolithiasis, particularly ChatGPT. In a recent single-model analysis, Çakır et al. evaluated ChatGPT’s ability to answer both patient-oriented and guideline-based questions about urinary stone disease, reporting that 95% of the answers were completely accurate and clinically acceptable [7]. Their study highlighted ChatGPT’s strengths in general and diagnostic topics, as well as its capacity to reproduce consistent responses. However, their evaluation was limited to a single LLM version and did not involve direct model-to-model comparison or expert consensus scoring. In contrast, our study expands on this foundation by systematically comparing four leading LLMs and incorporating domain-expert validation to ensure methodological robustness. Among the evaluated models, Claude demonstrated the highest correlation with expert consensus, suggesting a strong capability to generate guideline-consistent responses in the domain of urolithiasis. This performance may reflect architectural distinctions or training approaches that support structured medical information processing, as supported by recent evaluations showing Claude’s robust performance in diagnostic and decision-support tasks [12, 13]. The model’s relatively stable performance across various question themes may indicate potential utility in guideline-based informational support. Furthermore, the absence of significant differences in performance across the nine urolithiasis domains suggests that all LLMs evaluated in this study demonstrated relatively consistent performance across different thematic categories. This thematic stability may be relevant for future clinical and educational applications involving variable topic domains. Despite the promising results, several limitations of this study should be acknowledged. The use of standardized questions derived directly from guideline content creates a controlled test environment that does not fully reflect the variability and complexity of real-world clinical interactions. In practice, clinicians must navigate incomplete data, patient-specific nuances, and diagnostic uncertainty—factors that may challenge the real-world applicability of LLM outputs. Moreover, although expert scoring strengthens the study’s validity, subjective interpretation during consensus formation may introduce bias. Recent evidence also shows that model behavior can vary significantly based on prompt phrasing and clinical context, raising concerns about consistency and stability in live decision-making environments [14, 15]. These considerations underscore the importance of cautious implementation and ongoing evaluation of LLMs in real-time clinical settings. Real-world clinical decision-making involves individualized patient characteristics, imaging interpretation, laboratory findings, and evolving clinical scenarios that cannot be fully represented through standardized guideline-based questions alone. The results of this study underscore the emerging potential of LLMs as supportive tools in specialty-specific medical domains, particularly when benchmarked against established clinical guidelines. Their ability to generate accurate, consistent, and guideline-adherent responses suggests potential utility for future integration into clinical informational support systems. However, translating these capabilities into everyday practice will require further validation across diverse real-world scenarios, patient populations, and languages. Future research should focus on evaluating LLM performance within live clinical workflows, assessing user interaction dynamics, and exploring model integration with electronic health records. In parallel, regulatory frameworks and ethical standards must evolve to ensure responsible deployment, clinician oversight, and patient safety. With continued development, LLMs may become reliable assistants in urological care and beyond complementing but not replacing clinical expertise. Conclusion This study shows that LLMs, particularly Claude, can provide clinically consistent answers to urolithiasis guideline-based questions. Claude demonstrated the highest alignment with expert consensus. While model performance was stable across themes, real-world application will require further validation. LLMs may serve as adjunctive informational tools for guideline-based urological education and support, although further validation in real-world clinical settings remains necessary.