Evaluation of ChatGPT-4o’s and DeepSeek R1’s responses to urological problems: a comparative study Authors: Hanbo Lu, Yusa Zhang, Zhan Wang, Yang Zhao, Jiang Liu, Dongxu Qiu, Yushi Zhang Published: 2026 ABSTRACT AbstractBackgroundUrology presents unique challenges for AI systems, requiring both extensive medical knowledge and advanced reasoning. While large language models (LLMs) like GPT-4 have shown promise in medical education and decision support, their performance in urology remains underexplored.ObjectiveTo provide a time-stamped comparison of two representative large language models available at the time of evaluation, ChatGPT-4o and DeepSeek R1, in answering urology-related single-choice questions, and to evaluate their accuracy, stability, and response consistency across different response configurations.MethodsA total of 809 single-choice questions from the Chinese National Qualification Examination for Attending Physicians in Urology were administered to ChatGPT-4o and DeepSeek R1. Each model was tested under three configurations: basic mode, deep-thinking mode, and web-enabled retrieval. Accuracy was calculated for each configuration, and statistical comparisons were performed using McNemar’s test. Stability across reasoning modes was assessed by comparing performance variability. Additional analyses examined performance differences between short-answer and case-based clinical questions.ResultsChatGPT-4o achieved accuracy rates of 78.12%, 73.79%, and 78.99% in basic, deep-thinking, and web-enabled retrieval modes, respectively. DeepSeek R1 outperformed ChatGPT-4o across all configurations, with accuracy rates of 83.19%, 81.46%, and 84.55%, respectively. All between-model differences were statistically significant (p < 0.001). DeepSeek R1 demonstrated greater internal stability across reasoning modes, whereas ChatGPT-4o showed notable variability. In subgroup analyses, DeepSeek R1 exhibited a more pronounced advantage in complex, case-based clinical questions. Both models performed consistently across urological disease categories, and findings were limited to the Chinese-language context in which the evaluation was conducted.ConclusionDeepSeek R1 showed superior performance compared with ChatGPT-4o in both accuracy and stability when answering urology-related examination questions, particularly in complex case-based scenarios. These results suggest that optimized LLMs may have potential utility in urology education and examination-style question answering, especially within Chinese-language environments. However, these findings should not be interpreted as evidence of readiness for real-world clinical decision support, and further validation in clinically realistic settings is required.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1186/s12894-026-02228-9.Keywords: Urology, ChatGPT-4o, Deepseek, Large language model, Artificial Intelligence Background Urology, with its complex and diverse range of conditions, from urinary tract infections to urological cancers, presents a unique challenge for artificial intelligence (AI) systems [1, 2]. Accurate and reliable assistance in this field requires not only vast medical knowledge but also advanced reasoning abilities to interpret nuanced clinical scenarios. Previous studies have demonstrated the potential of large language models (LLMs) in answering medical questions [3]. For example, research in radiology showed that GPT-4 achieved over 80% accuracy in board-exam style questions [4], and in dentistry, LLMs were able to correctly answer more than two-thirds of national specialty examination questions [5]. Recent urology-specific studies have begun to compare AI-generated responses in clinically oriented question-answering tasks. However, systematic and cross-model comparisons across specialized medical disciplines remain scarce. Such comparisons may help medical professionals better understand the potential and limitations of these tools. With the rapid development of AI, LLMs such as OpenAI’s ChatGPT have shown promise in medical education [7], diagnostic reasoning, and decision support [8]. Extant research has examined the performance of LLMs in standardized medical licensing examinations, including the United States Medical Licensing Examination (USMLE) [9], the American Board of Anesthesiology (ABA) examinations [10], the Japanese Medical Licensing Examination (JMLE) [11], and the Chinese National Medical Licensing Examination (CNMLE) [12]. These models can process complex clinical information and generate human-like responses, enabling novel applications in both learning and practice. In parallel, emerging models like DeepSeek R1 have been developed and fine-tuned with domain-specific data, offering potentially superior performance in non-English or specialty-specific contexts [13–15]. Despite these advances, the comparative effectiveness of LLM in specialized fields like urology remains underexplored. To address this gap, the present study conducts a head-to-head evaluation of ChatGPT-4o and DeepSeek R1 across a comprehensive set of 809 urology-related single-choice questions from the Chinese National Qualification Examination for Attending Physicians in Urology. By comparing their accuracy, reasoning stability, test–retest stability, and performance across different urological domains and question types, we aimed to assess their potential utility as supportive tools for urology education and examination-style knowledge assessment. Key Words: Urology, ChatGPT-4o, Deepseek, Large language model, Artificial Intelligence Methods Study design This was a cross-sectional, in silico evaluation designed to compare the performance of two large language models (LLMs), ChatGPT-4o and DeepSeek R1, in solving urology-related single-choice questions. The evaluation workflow is illustrated in Fig. 1. A total of 809 standardized single-choice questions in urology were submitted to both models in three configurations: basic mode, advanced reasoning mode (deep-thinking), and web-enabled retrieval. The advanced reasoning condition for each model was operationally defined as the platform-designated deep-thinking option available via the consumer web interface. This setting encourages the model to generate more structured, stepwise, or chain-of-thought style responses, potentially improving accuracy on complex questions. Importantly, this option does not modify the internal model weights, architecture, or knowledge base. Because the underlying implementation is proprietary and may vary across sessions or platform updates, this condition is treated as an interface-enabled configuration rather than a controlled model-level manipulation. All six model configurations were evaluated independently under controlled and isolated testing conditions. To reduce the risk of potential test-set contamination, the question set was selected from a urology-specific examination source that was not publicly available as a complete dataset on the open Internet at the time of evaluation. We also performed manual checks using representative question stems to assess whether the items were directly searchable online. However, because proprietary LLM training datasets are not fully transparent and some public Chinese medical examination corpora exist, potential overlap with the test questions cannot be completely excluded. Each model received the same set of questions, in the same format, without prompts, feedback, or iterative interactions to simulate real-world usage scenarios. Each model configuration was evaluated in three independent runs using the same set of 809 questions. For the primary accuracy analysis, accuracy was calculated separately for each run and then summarized across the three runs as the mean accuracy. Statistical comparisons between models were performed at the item level within corresponding runs, and the reported results represent the average paired performance across repeated evaluations. Separately, test–retest stability was assessed by comparing whether the model produced the same answer for each question across the three runs. A question was defined as unstable if at least one of the three responses differed from the others. This study was designed as a time-stamped benchmark of selected representative LLMs rather than a comprehensive ranking of all available models. All evaluations were conducted between 4/22/2025 and 5/27/2025, and the results should be interpreted in relation to the model versions and platform settings available during this evaluation window. Workflow of Model Evaluation in Urological Question Answering. A total of 809 urology-related questions were collected from Chinese National Qualification Examination for Attending Physicians. Six model configurations, including three versions of ChatGPT-4o and three of DeepSeek R1 (basic, deep-thinking, and web-enabled retrieval), were tested. Model performance was evaluated based on overall accuracy, response stability and accuracy between different question types Data source All 809 questions were derived from the National Qualification Examination for Attending Physicians in Urology, ensuring that they reflect standard clinical knowledge and practice guidelines in China. Each item was accompanied by an officially released standard answer, guaranteeing the accuracy and reliability of the reference solutions. The dataset covered a wide spectrum of urological topics, including: Genitourinary tuberculosis (91 questions); Urolithiasis (69 questions); Urological trauma (90 questions); Urinary tract infections (116 questions); Urinary dysfunction (43 questions); Adrenal gland disorders (92 questions); Obstructive uropathy (68 questions); Congenital anomalies (73 questions) and Urological cancers (167 questions). Additionally, the questions were stratified into short-answer format (n = 523) and case-based clinical scenarios (n = 286). A case-based question presents a clinical scenario that requires diagnostic reasoning and management decision-making. For instance: A 35-year-old man presented with a painless right testicular mass for 2 months. Physical examination revealed an enlarged right testis with a palpable hard mass and a feeling of heaviness; the transillumination test was negative. Scrotal ultrasonography suggested a right testicular tumor. Chest X-ray showed no space-occupying lesions, while abdominopelvic CT indicated retroperitoneal lymph node metastasis. After a right radical orchiectomy, the pathological diagnosis was seminoma. What should be the next step in management? A.Radiotherapy Radiotherapy B.Close follow-up observation Close follow-up observation C.Combination chemotherapy Combination chemotherapy D.Radiotherapy followed by chemotherapy Radiotherapy followed by chemotherapy E.Retroperitoneal lymph node dissection (Correct answer: A. Radiotherapy.) Retroperitoneal lymph node dissection (Correct answer: A. Radiotherapy.) In contrast, a short-answer question focuses on factual recall or key conceptual understanding, such as: Testicular tumors most commonly occur in young and middle-aged men. The diagnosis is based on the following key points, except: A.Unilateral testicular enlargement or palpable mass, smooth surface, firm consistency, nonelastic, and feeling of heaviness Unilateral testicular enlargement or palpable mass, smooth surface, firm consistency, nonelastic, and feeling of heaviness B.Elevated β-HCG, AFP, and lactate dehydrogenase (LDH) Elevated β-HCG, AFP, and lactate dehydrogenase (LDH) C.Ultrasonography and CT examination Ultrasonography and CT examination D.MRI examination MRI examination E.Intermittent hematospermia (Correct answer: E. Intermittent hematospermia.) Intermittent hematospermia (Correct answer: E. Intermittent hematospermia.) AI models Six large language model (LLM) configurations were evaluated in this study, encompassing two base models, ChatGPT-4o (https://platform.openai.com/docs/models/chatgpt-4o-latest) and DeepSeek R1 (https://github.com/deepseek-ai/DeepSeek-R1), with three different response modes for each. The tested configurations were: ChatGPT-4o (Basic Mode) ChatGPT-4o (Basic Mode) ChatGPT-4o with Advanced Reasoning Mode (ChatGPT-4o deep-thinking) ChatGPT-4o with Advanced Reasoning Mode (ChatGPT-4o deep-thinking) ChatGPT-4o with web-enabled retrieval ChatGPT-4o with web-enabled retrieval DeepSeek R1 (Basic Mode) DeepSeek R1 (Basic Mode) DeepSeek R1 with Advanced Reasoning Mode (DeepSeek R1 deep-thinking) DeepSeek R1 with Advanced Reasoning Mode (DeepSeek R1 deep-thinking) DeepSeek R1 with web-enabled retrieval DeepSeek R1 with web-enabled retrieval In basic mode, questions were submitted directly without enabling external web search or retrieval functions. In deep-thinking mode, the platform-designated reasoning or deep-thinking option was selected when available. In retrieval-augmented mode, the web-enabled retrieval function provided by the platform was enabled, allowing the model to consult external online information before generating an answer. Therefore, comparisons between the retrieval/search-augmented mode and the non-retrieval modes should be interpreted as comparisons between an open-book, externally assisted setting and closed-book settings based on the models’ internal knowledge. These comparisons were not intended to isolate the intrinsic reasoning ability of the base model alone. For each configuration, all 809 urological single-choice questions were submitted individually through the official consumer web interfaces of the corresponding platforms during the predefined evaluation period (4/22/2025 and 5/27/2025). The original question stem and answer options were copied and pasted without modification, and no additional prompts, contextual information, hints, feedback, or formatting changes were provided. Each question was entered in a new isolated session to minimize memory leakage or carryover effects. Because consumer web interfaces rather than controlled programmatic application programming interfaces were used, generation parameters such as temperature, top-p, maximum output length, and decoding strategy could not be directly controlled or independently verified. In addition, platform-level factors, including hidden system instructions, potential model routing, interface-dependent behavior, and automatic product updates, were not fully transparent. The model names and available interface options were recorded as displayed by each platform at the time of testing. The outputs were recorded verbatim. For scoring, the final explicitly stated answer option was extracted as the model’s response. If the model generated explanatory text before selecting an option, only the final selected option was used for scoring. If multiple options were mentioned but one option was clearly identified as the final answer, that option was recorded. If no unambiguous final answer could be determined, the response was classified as incorrect [16]. Statistical analysis Statistical analyses were conducted to compare the accuracy of the six AI model configurations across 809 urological single-choice questions. Accuracy was defined as the proportion of correct responses. When comparing different models evaluated on the same set of questions, McNemar’s test was used to account for paired binary outcomes. When comparing the performance of the same model across different question categories (e.g., short-answer vs. case-based) or disease subgroups, where observations were independent, Pearson’s chi-square (χ²) test was applied. Effect sizes and 95% Clopper-Pearson confidence intervals (CIs) were calculated for accuracy estimates for each model and question type. Paired accuracy differences between models were also computed, with 95% confidence intervals estimated to assess the magnitude and precision of performance gaps. A two-sided p-value < 0.05 was considered statistically significant. All analyses were performed using Python (version 3.12.7) with the pandas, numpy, and scipy.stats libraries, and visualizations were generated using matplotlib and seaborn. Results Comparative accuracy and robustness across disease categories As shown in Fig. 2-A, DeepSeek R1 demonstrated consistently higher accuracy across all three operational modes compared with ChatGPT-4o. In the basic mode, DeepSeek R1 achieved an accuracy of 83.19%, significantly outperforming ChatGPT-4o (78.12%, p < 0.001). In the deep-thinking mode, DeepSeek R1 reached 81.46%, exceeding ChatGPT-4o’s 73.79% (p < 0.001). Similarly, in the web-enabled retrieval mode, DeepSeek R1 obtained the highest overall accuracy of 84.55%, compared with 78.99% for ChatGPT-4o (p < 0.001). Comparative Performance of ChatGPT-4o and DeepSeek R1 Models on Urological Question Sets. (A) Model Accuracy Comparison. Bar plots showing mean accuracy (%, y-axis) of ChatGPT-4o and DeepSeek R1 under three configurations (Basic, Deep-thinking, and web-enabled retrieval). Error bars indicate Clopper-Pearson 95% confidence intervals. All pairwise differences between models were statistically significant (p < 0.001, McNemar’s test). (B) Paired Accuracy Difference with 95% Confidence Intervals. Forest-style plot illustrating the paired differences (ChatGPT-4o − DeepSeek R1) with corresponding 95% confidence intervals. Negative values indicate higher accuracy for DeepSeek R1 across all configurations. (C) Accuracy of Each Model on Different Urological Disease Categories. Line plots depicting the variation of accuracy across nine urological domains (x-axis). Each color represents one model configuration, and the p-values refer to within-model differences across disease categories (Chi-square test). (D) Accuracy Heatmap Across Diseases and Model Configurations. Heatmap summarizing model performance (accuracy %) across disease categories (x-axis) and model types (y-axis). Darker red shades indicate higher accuracy The paired accuracy differences with 95% confidence intervals are shown in Fig. 2-B, where all intervals exclude zero (Basic: − 6.76% to − 3.37%; Deep-thinking: − 9.71% to − 5.62%; web-enabled retrieval: − 7.45% to − 3.68%), indicating statistically robust and practically meaningful improvements. Overall, these results confirm a consistent performance advantage of DeepSeek R1 across all configurations. Across the nine urological disease categories, each model demonstrated relatively consistent accuracy (Fig. 2-C). No significant inter-disease variation was observed for any single model (p > 0.05), indicating stable generalization across diverse diagnostic topics. While minor fluctuations existed, DeepSeek R1-web-enabled retrieval maintained both high and stable accuracy across diseases, whereas the basic versions of both models exhibited wider variability.The heatmap visualization in Fig. 2-D further depicts detailed performance patterns across model configurations and disease types. DeepSeek R1 exhibits superior intra-model stability and consistency across multiple reasoning modes and repeated tests When evaluating performance consistency across different reasoning modes, DeepSeek R1 demonstrated notably greater stability in accuracy compared with ChatGPT-4o. The 95% confidence intervals for DeepSeek R1 across its three modes (Basic, Deep-thinking, and web-enabled retrieval) were narrower and showed substantial overlap, suggesting that its performance remained steady regardless of reasoning strategy. In contrast, ChatGPT-4o exhibited wider confidence intervals and greater dispersion, indicating higher variability in accuracy between modes (Fig. 3-A). Stability of Accuracy Across Reasoning Modes and Repeated Testing Within Each Model. (A) DeepSeek R1 demonstrated markedly higher intra-model stability than ChatGPT-4o, as reflected by narrower and overlapping 95% confidence intervals across reasoning modes (basic, deep-thinking, and web-enabled retrieval). (B) In the repeated testing of the same 809 urology-related questions, DeepSeek R1 produced significantly fewer unstable responses across all modes, confirming superior test–retest reliability(McNemar’s test). The 95% confidence intervals of paired accuracy differences between ChatGPT-4o and DeepSeek R1 remained consistently below zero across all reasoning modes, further supporting the robustness of DeepSeek R1’s performance Notably, this stability was consistent even in the repeated testing of the same 809 urology-related questions. To assess test–retest stability, each model independently answered the entire question set three times under identical conditions. A question was considered unstable if any of the three responses differed. The proportion of such unstable questions was then calculated and compared between models across reasoning modes. Specifically, when evaluating the model’s accuracy across three separate trials, DeepSeek R1 demonstrated significantly fewer unstable questions. For example, in the basic mode, DeepSeek R1 exhibited only 3.8% unstable answers compared to 6.1% for ChatGPT-4o (p = 0.0021). This pattern held across all reasoning modes—Deep-thinking (3.7% vs. 6.3%, p = 0.0005) and web-enabled retrieval (3.2% vs. 5.8%, p = 0.0003)—further reinforcing the superior stability of DeepSeek R1. As shown in the paired comparison, the 95% confidence intervals of accuracy differences between ChatGPT-4o and DeepSeek R1 remained consistently below zero across all reasoning modes, with mean paired differences ranging from − 2.9% to − 3.7% (Fig. 3-B). DeepSeek R1 Outperforms ChatGPT-4o more significantly in case-based questions, highlighting superior reasoning capabilities To evaluate model performance across different question types, short-answer and case-based items were analyzed separately. For short-answer questions, DeepSeek R1 achieved accuracies of 82.4%, 80.7%, and 84.5% under the basic, deep-thinking, and web-enabled retrieval modes, whereas ChatGPT-4o scored 78.2%, 74.6%, and 80.5%. For case-based questions, DeepSeek R1 achieved 84.6%, 82.9%, and 84.6%, compared with ChatGPT-4o’s 78.0%, 72.4%, and 76.2%. When viewed through the 95% confidence interval plots, the differences are visually evident: for short-answer questions, the intervals are close, indicating a small difference between models, whereas for case-based questions, the intervals are clearly separated, highlighting a substantial performance gap in favor of DeepSeek R1 (Fig. 4-C). Comparison of Model Accuracy Across Question Types and AI Architectures.(A) Accuracy of six models on short-answer and case-based medical questions. Bars represent mean accuracy ± standard error. No statistically significant differences were observed between short-answer and case-based formats within the same model (all p > 0.05, Chi-square test). (B) Heatmap showing detailed accuracy values for each model and question format. (C) Side-by-side comparison of ChatGPT-4o and DeepSeek R1 performance across three reasoning modes (Basic, Deep-thinking, and web-enabled retrieval) for short-answer and case-based questions. While DeepSeek R1 consistently outperformed ChatGPT-4o across all modes, the performance gap was more pronounced for case-based questions Within-model comparisons showed no statistically significant differences between performance on short-answer and case-based questions for either model (Fig. 4-A, B). However, DeepSeek R1 consistently outperforms ChatGPT-4o across question types, and its advantage is more pronounced in case-based items. This suggests a potential strength in handling context-rich, clinically oriented reasoning tasks. This outcome may result from the differences in training data sources and model tuning. Discussion Principal findings In recent years, large language models (LLMs) have shown great promise in supporting clinical decision-making, medical education, and diagnostic reasoning [7]. Several studies have evaluated the reasoning capabilities of models like GPT-4 in various medical domains [7, 17], including general internal medicine [3], oncology [18], and radiology [19]. However, there is a lack of large-scale, domain-specific benchmarking focused on urology, especially one that compares different operational modes (basic, deep-thinking, and web-enabled retrieval) across multiple LLM platforms. In this study, we systematically evaluated six model configurations, ChatGPT-4o and DeepSeek R1, each in basic, deep-thinking, and web-enabled retrieval modes, across 809 urology-related questions spanning nine disease categories. Our findings demonstrate that DeepSeek R1 consistently outperforms ChatGPT-4o in overall accuracy, with DeepSeek R1-web-enabled retrieval achieving the highest score (84.55%). DeepSeek R1 also exhibited superior internal stability across different reasoning modes, in contrast to ChatGPT-4o which showed significant variability. Besides, during repeated tests, DeepSeek R1 demonstrated greater stability than ChatGPT-4o across all modes. Importantly, while both models performed stably across different urological disease types, DeepSeek R1 showed more pronounced advantages in complex, context-rich case-based questions. Interpretation and strengths Specifically, we show that while ChatGPT-4o performs well overall, DeepSeek R1—especially with web-enabled retrieval enabled—achieves higher and more stable accuracy in urology, a field that demands nuanced reasoning and contextual understanding. However, the performance advantage observed in the retrieval-augmented mode should be interpreted cautiously because this setting allowed access to external online resources, whereas the basic and deep-thinking modes were closed-book conditions. Therefore, the difference between retrieval/search-augmented and non-retrieval modes reflects a combination of model capability, retrieval access, and the quality of externally retrieved information, rather than intrinsic reasoning ability alone. Our subgroup analysis revealed that DeepSeek R1’s advantage is particularly evident in case-based clinical questions, indicating stronger capabilities in multi-step reasoning and decision-making. This may stem from differences in training data coverage, retrieval strategies, and reasoning architecture between the two models. Compared with prior benchmarking studies on English-language medical examinations [7, 20], where Med-PaLM 2 achieved near-physician-level performance with up to 86.5% accuracy on medical exam datasets and GPT-4 outperformed GPT-3.5 on the Polish Medical Final Examination with an average accuracy of 79.7%, the performance observed in our study (78–85%) appears lower. However, these differences likely reflect the distinct dataset characteristics and linguistic complexity rather than model limitations. Importantly, our dataset—derived from the Chinese National Qualification Examination for Attending Physicians in Urology—represents a large-scale, high-quality corpus aligned with real clinical education and practice in China, which has not previously been evaluated as a urology-specific benchmark in LLM research. Nevertheless, because publicly available Chinese medical examination corpora exist and the proprietary LLM training data are not fully disclosed, partial overlap with model training data cannot be fully ruled out. As such, our findings offer valuable insight into how general-purpose LLMs perform under authentic Chinese-language conditions, providing a more practically relevant benchmark for large-scale application and localized optimization in Chinese medical contexts. Our study also presents several advantages over previous work. First, we used the largest urology-specific question set to date, covering a wide range of clinically relevant subdomains. Second, we employed multiple operational modes for each model, allowing for a fine-grained comparison of reasoning strategies. Third, we incorporated statistical comparisons not only across models but also across disease types and question formats, offering a more comprehensive and multidimensional evaluation framework. Clinical and technical implications Differences between ChatGPT-4o and DeepSeek R1 have important implications for clinical practice [21] and education in urology. In residency training or standardized examination preparation, a more accurate and stable model may support automated question-bank tutoring, explanation generation, and self-assessment. However, because the present evaluation was based on single-choice examination questions rather than real patient data, multi-turn clinical encounters, or prospective clinical workflows, the results should not be interpreted as direct evidence that these models are ready for autonomous clinical decision support. Any potential clinical use would require additional validation using real-world clinical cases, open-ended reasoning tasks, and clinician-supervised prospective evaluation. Regarding model implementation, each model was evaluated under three built-in operational configurations—basic, deep-thinking, and web-enabled retrieval—as provided by their respective platforms. These modes were selected within the model interface to represent varying levels of reasoning depth and knowledge retrieval integration. The web-enabled retrieval configuration, in particular, enables the model to reference external or pre-indexed knowledge sources during reasoning, which may explain its superior performance in complex, case-based questions. Limitations However, several limitations should be acknowledged. First, although the question set was large and diverse, it originated from a single national platform, which may introduce regional or linguistic bias and limit generalizability beyond China [23, 24]. Nevertheless, as fundamental principles of medical reasoning are largely universal, this influence on overall validity is likely minimal [25]. Second, the evaluation involved only single-choice questions and structured scenarios, which cannot fully represent real-world diagnostic reasoning [26]. This design may underestimate models’ reasoning depth but also reduces the risk of open-ended hallucinations. Future studies incorporating open-ended prompts or multi-answer questions would provide a more comprehensive assessment [27]. Third, although DeepSeek R1 (web-enabled retrieval mode) achieved an accuracy of about 85%—above both the national passing threshold (60%) and average physician scores (68.7%)—direct comparison with human experts using external resources (“Human web-enabled retrieval”) remains lacking due to the absence of relevant data from proctored examinations. Fourth, while the single-choice format inherently minimizes the risk of hallucination by constraining possible outputs, it does not eliminate the issue entirely. Standardized frameworks for quantitative hallucination evaluation are still urgently needed to ensure the safe and trustworthy clinical application of medical LLMs, particularly when models are used in open-ended or decision-support contexts [28]. Another limitation is the time-sensitive nature of LLM benchmarking. ChatGPT-4o and DeepSeek R1 were selected as representative models available at the time of evaluation, but LLM capabilities evolve rapidly and newer proprietary or open-weight models may perform differently. Therefore, the present findings should be interpreted as a time-stamped comparison of selected models rather than a definitive assessment of the current state of all LLMs. Future studies should update this benchmark with newer models and controlled programmatic interfaces. What’s more, we did not prospectively register adherence to TRIPOD-LLM, which we acknowledge as a reporting limitation. Some items of TRIPOD-LLM, such as patient-level prospective validation, were not directly applicable to this study, which focused on evaluating LLM performance on structured urology examination questions. Therefore, while we followed the general principles of transparent reporting where possible, not all checklist items were implemented [29]. Ethical considerations The integration of large language models (LLMs) into clinical and educational settings raises important ethical considerations [30]. Although both ChatGPT-4o and DeepSeek R1 demonstrated high accuracy, the observed performance gaps highlight the need for cautious deployment, particularly in scenarios involving clinical decision-making. Issues such as potential bias, overreliance on AI-generated outputs, and the absence of clear accountability frameworks must be addressed before widespread implementation. Ensuring transparency in model design, maintaining clinician oversight, and establishing rigorous validation protocols will be essential to safeguard patient safety and professional integrity [31]. Future directions Future research should expand the evaluation to other languages and true clinical data, integrate open-ended and multi-turn interactions, and explore alignment strategies to further improve clinical safety. Additionally, investigating user-centered metrics such as trustworthiness, interpretability, and integration into real clinical workflows would be valuable [32, 33]. As LLMs continue to evolve, rigorous domain-specific benchmarking—such as the one we present here—will be critical for informing safe and effective AI deployment in medical practice. Conclusion Both ChatGPT-4o and DeepSeek R1 maintained consistent performance across diverse urological disease categories without significant disease-specific fluctuations. However, compared with ChatGPT-4o, DeepSeek R1-web-enabled retrieval achieved the higher overall accuracy and greater stability, especially in handling complex, context-rich medical reasoning tasks. It should be noted that our evaluation was limited to Chinese-language data, and the generalizability of these findings to other countries remains to be validated. These findings suggest that DeepSeek R1, especially in web-enabled retrieval mode, may offer more reliable and accurate assistance in urology-related AI applications. Supplementary Information Supplementary Material 1. Acknowledgements None. Authors’ contributions Conceptualization, supervision and funding acquisition: YSZ; Writing original draft and data curation: HBL; Review and editing: DXQ; Visualization: YSZ; Resources: ZW, YZ, JL. Funding Postdoctoral Fellowship Program of CPSF. (Grant Number: GZC20230301);Beijing Natural Science Foundation (L258065); CAMS Innovation Fund for Medical Sciences(CIFMS)(2024-I2M-C&T-B-023). Natural Science Foundation of Hunan Province (2026JJ70064) , Research Foundation of Education Bureau of Hunan Province, China(Grant No.25C0013 ), and the funding from Hunan Provincial People's Hospital (2025YJKT13) Data availability The datasets used and/or analyzed during the current study are available from the corresponding author on request. Declarations Ethics approval and consent to participate Not applicable. This study did not involve human participants, human data, human tissue, animal experiments, patient records, medical images, or identifiable personal information. The analysis was based solely on pre-existing urology examination questions used for educational assessment and responses generated by large language models. No individual-level data from examinees, patients, clinicians, or platform users were collected, processed, or analyzed. Therefore, ethics approval and informed consent were not required. Consent for publication Not applicable. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Hanbo Lu and Yusa Zhang contributed equally to this work. Change history 9/27/2026 The original online version of this article was revised: The affiliation for the corresponding author, Dongxing Qiu, was incorrect and has now been updated. Contributor Information Dongxu Qiu, Email: qiudongxu1996@163.com. Yushi Zhang, Email: beijingzhangyushi@126.com. References Associated Data Supplementary Materials Supplementary Material 1. Data Availability Statement The datasets used and/or analyzed during the current study are available from the corresponding author on request.