From accuracy to robustness: a comparative study of five advanced large language models on the Chinese dental licensing examination Authors: Jiarui Zhang, Pascal Ubuzima, Gaoyang Huang, En-Hsuan Lee, Yuting Wang, Huifang Xu, Lei Xia, Tingting Wu Published: 2025 ABSTRACT BackgroundLarge language models (LLMs) have demonstrated considerable promise in various domains, including dentistry. This study aimed to evaluate five advanced LLMs (DeepSeek-R1, GPT-4o, OpenAI o3, GPT-5 Thinking, and Gemini 2.5 Pro) in the context of the Chinese Dental Licensing Examination (CDLE) to explore their potential in dental education and practice.MethodsA total of 600 questions were selected from the official review book provided by the Chinese National Medical Examination Center. All questions, presented in Chinese, were submitted individually to the five LLMs via their web interfaces. The responses were classified as “correct” or “incorrect” using the official answer keys provided by the review book. We analyzed and compared each model’s overall accuracy and accuracy across different subjects and question types using χ2 or Fisher’s exact tests, as appropriate. To assess robustness, 120 of the 600 questions were selected for adversarial testing under two types of perturbations. We employed McNemar’s test to measure each model’s accuracy degradation during adversarial testing.ResultsDeepSeek-R1, GPT-5 Thinking, Gemini 2.5 Pro, and OpenAI o3 demonstrated superior performance, significantly surpassing GPT-4o (p < 0.001), with Gemini 2.5 Pro achieving the highest accuracy at 91.67%. Performance varied across dentistry and its sub-disciplines (prosthodontics and oral anatomy), where GPT-4o significantly lagged behind the other four LLMs (p < 0.05). Gemini 2.5 Pro and GPT-5 Thinking outperformed GPT-4o on A1 and B1 question types (p < 0.05). In adversarial testing, all LLMs showed a slight decrease in accuracy, ranging from 1.66% to 5.84%, but the drop was not significant (p > 0.05).ConclusionsUsing the CDLE as a benchmark, new-generation LLMs achieved markedly higher accuracy. Furthermore, all models exhibited strong robustness against adversarial perturbations. These findings indicate that advanced LLMs hold promise as assistive tools for dental education and practice.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12903-025-07540-8. Background Large language models (LLMs), exemplified by OpenAI’s GPT series and Google’s Gemini series, have catalyzed a global wave of advances in artificial intelligence, demonstrating impressive abilities in natural language understanding and reasoning [1]. These models hold immense promise for the dental field, with potential applications in education, diagnostics, and patient care [2–4]. However, a rigorous assessment of their performance and safety is essential before their deployment in dental education and practice. The Chinese Dental Licensing Examination (CDLE) is organized by the Chinese National Medical Examination Center (CNMEC) and serves as a key benchmark for assessing candidates’ professional abilities [5]. The exam contains 600 multiple-choice questions (MCQs), each with a single correct answer. To be eligible for the CDLE, candidates must possess a bachelor’s degree or higher in the dental field and complete a full year of practical clinical work in a medical institution under the supervision of a licensed physician. From 2020 to 2023, the average pass rate for the CDLE was only 50% [6], underscoring the ongoing need for educational resources to support candidates. Recent studies have demonstrated that LLMs have achieved relatively high accuracy in dental examinations, covering areas like head and neck anatomy, oral pathology, and periodontology [7–9]. This suggests that LLMs have preliminarily established the specialized knowledge base to support dental education and practice. However, as Nori et al. have noted, high accuracy does not necessarily guarantee reliability in complex clinical settings [10]. Unlike standardized examinations with clear and structured questions, patient complaints and medical records in real-world scenarios often contain ambiguous and redundant information. Medical students may also provide imprecise descriptions and use colloquial wording when asking questions. Even if a model achieves high accuracy on standardized examinations, excessive sensitivity to minor wording changes or irrelevant information may lead to unstable or misleading outputs in practice. Therefore, accuracy alone is insufficient to measure a model’s actual competence. One key approach to explore how LLMs perform in more realistic scenarios is to evaluate their robustness, the ability to maintain stability despite input disturbances [11]. Robustness is not merely a technical metric; it is a prerequisite for deploying LLMs in dental education and practice. Although Hu et al. and Xiong et al. have preliminarily explored the performance of LLMs on the CDLE [5, 6], their studies were limited to earlier model versions and did not cover the newly released models with stronger reasoning capabilities. Moreover, previous evaluations have primarily centered on accuracy, whereas robustness evaluations in dental contexts remain relatively limited [12]. To bridge this gap, this study aims to assess the accuracy and robustness of five representative advanced LLMs (DeepSeek-R1, GPT-4o, OpenAI o3, Gemini 2.5 Pro, and GPT-5 Thinking) using the CDLE as a benchmark. These models were released between 2024 and 2025, covering the latest architectures and reasoning paradigms from leading technology companies. To our knowledge, this is the first evaluation of the new versions of LLMs on the CDLE. This research not only addresses a significant gap in the existing literature but also offers valuable perspectives for the optimization of LLMs in the realms of dental education and practice. Methods Ethical considerations This study was conducted at Anhui Medical University and exempted from ethics review due to the use of publicly available data. Models evaluated GPT-4o (OpenAI, San Francisco, CA, USA, released on May 13, 2024) is the “omni” version of the GPT-4 family, built on an end-to-end multimodal transformer architecture. It is OpenAI’s first unified model that support text, image, and audio inputs and outputs [13, 14]. OpenAI o3 (OpenAI, released on April 16, 2025) is the successor to OpenAI o1, characterized by its specialized focus on enhanced reasoning capabilities. Trained via large-scale reinforcement learning to elicit chains-of-thought (CoT), OpenAI o3 exhibits high performance in solving complex math, coding, and scientific challenges [15, 16]. GPT-5 Thinking (OpenAI, released August 7, 2025) is a deep-reasoning multimodal transformer model within the unified GPT-5 system. As the successor to OpenAI o3, it can allocate additional internal reasoning computation on demand to improve answer quality and robustness [17, 18]. Gemini 2.5 Pro (Google DeepMind, Mountain View, CA, USA, released on June 17, 2025) is an advanced reasoning model in the Gemini 2.5 series. It is a multimodal model employing a mixture-of-experts (MoE) transformer architecture, designed for high-complexity tasks such as sophisticated reasoning, code generation, ultra-long context processing, and multimodal understanding [19]. DeepSeek-R1 (DeepSeek, Hangzhou, China, released on January 20, 2025) is the first-generation reasoning model released by DeepSeek, based on the MoE architecture with 671 billion parameters and 37 billion active parameters. Trough reinforcement learning, it improves performance in mathematics, code generation, and logical CoT processes, representing a prominent example of open-source LLMs [20, 21]. Data collection Because the CNMEC does not provide past CDLE exam questions, we constructed our dataset from the latest official review book (2025 edition) provided by the CNMEC, in accordance with the 2025 examination syllabus. The review book is publicly available for purchase through the CNMEC’s official website, and no special permissions or formal authorization are required to use the materials. To ensure the question types and subject distribution in the dataset closely mirror the actual CDLE exam, we established a stratified sampling framework based on question type and discipline. The sample sizes for each stratum were determined according to the proportions outlined in the exam syllabus. Then, within each stratum, simple random sampling was conducted using computer-generated random numbers. Ultimately, 600 MCQs were included (see Additional file 5). Considering DeepSeek-R1 does not support multimodal input, we excluded visual content and only selected text-based questions to ensure a consistent baseline for comparison. The selected questions cover five primary disciplines: basic medicine (n = 39), clinical medicine (n = 39), medical humanities (n = 48), preventive medicine (n = 9), and dentistry (n = 465), as detailed in Table S1 (see Additional file 1). Given that dentistry accounts for the largest proportion of questions, we further subdivided it into ten sub-disciplines to conduct a more comprehensive evaluation. The ten sub-disciplines of dentistry were as follows: maxillofacial medical imaging diagnostics (n = 9), oral and maxillofacial surgery (n = 114), oral mucosal diseases (n = 21), pediatric dentistry (n = 21), periodontology (n = 39), prosthodontics (n = 112), endodontics (n = 45), oral anatomy (n = 29), oral pathology (n = 27), and oral preventive medicine (n = 48), as shown in Table S2 (see Additional file 2). These questions comprise four question types: A1 (Single-Sentence Best-Choice Question, n = 304), A2 (Case Summary Best-Choice Question, n = 121), A3/A4 (Case Group Best-Choice Question, n = 89), and B1 (Standard Matching Question, n = 86), as shown in Table S3 (see Additional file 3) [20, 22]. A1 and B1 question types, collectively referred to as knowledge-based questions, primarily assess memory and understanding of factual knowledge, whereas A2 and A3/A4 questions, collectively termed as case analysis questions, evaluate logical reasoning and comprehensive analysis of clinical scenarios [23]. Study design Before the study, we registered new accounts for each LLM and accessed them through the official web interfaces. From August 6 to 11, 2025, an independent researcher (E.L.) inputted the questions in Chinese sequentially into the five LLMs and recorded the output responses using Microsoft Excel 2021 (Microsoft Corp, Redmond, WA, USA). The process did not utilize any APIs or third-party plugins. All models were operated under default platform parameters such as temperature, without applying designed CoT prompts or activating enhanced features such as extensive reasoning. To eliminate the impact of historical conversations on model outputs, we cleared cookies and browsing history prior to each query and ensured that each question was posed in a completely new conversation window. Each question was submitted once, and the models were required to provide a single, clear answer, which was recorded immediately. To standardize prompts across platforms, we employed a role-playing strategy, instructing the LLMs to act as medical experts in the relevant disciplines. The general prompt template was as follows: “You are an expert in [discipline name]; please select the most appropriate answer from the following options.” For example, for a question related to endodontics, the full input was as follows:You are an expert in endodontics; please select the most appropriate answer from the following options.A 12-year-old patient presents with occlusal caries of the mandibular left first molar. During cavity preparation, an accidental pulp exposure approximately the size of a needle point occurs. The most appropriate management is:ADirect pulp cappingBIndirect pulp cappingCVital pulpotomyDPulp mummificationERoot canal treatment You are an expert in endodontics; please select the most appropriate answer from the following options. You are an expert in endodontics; please select the most appropriate answer from the following options. A 12-year-old patient presents with occlusal caries of the mandibular left first molar. During cavity preparation, an accidental pulp exposure approximately the size of a needle point occurs. The most appropriate management is:ADirect pulp cappingBIndirect pulp cappingCVital pulpotomyDPulp mummificationERoot canal treatment A 12-year-old patient presents with occlusal caries of the mandibular left first molar. During cavity preparation, an accidental pulp exposure approximately the size of a needle point occurs. The most appropriate management is:ADirect pulp cappingBIndirect pulp cappingCVital pulpotomyDPulp mummificationERoot canal treatment ADirect pulp capping Direct pulp capping BIndirect pulp capping Indirect pulp capping CVital pulpotomy Vital pulpotomy DPulp mummification Pulp mummification ERoot canal treatment Root canal treatment Upon completion of data collection, two researchers (Y.W. and H.X.) verified the correct answers using the official answer keys provided by the review book (2025 edition). Any discrepancies were resolved through discussion and a joint re-examination of the original questions. Unresolved disagreements were referred to a third senior researcher (T.W.) for arbitration, whose decision was accepted as final. To assess robustness, we conducted adversarial testing. The sample size estimation was performed using G*Power 3.1.9.7 (University of Düsseldorf, Düsseldorf, Germany). Previous robustness studies suggest that the introduction of perturbations often results in a 10%–20% drop in model accuracy [24, 25]. Based on this evidence, we set the parameters as follows: significance level (α) = 0.05, test power (1–β) = 0.8, preset odds ratio (OR) = 0.33, and discordant pairs ratio = 0.2. The analysis indicated that at least 115 questions were required. To strengthen the rigor of the findings, we ultimately selected 120 of the 600 questions using stratified random sampling (see Additional file 6). We then employed Claude Sonnet 4 (Anthropic, San Francisco, CA, USA, released on May 23, 2025) to rewrite these questions. This external model was chosen for two primary reasons. First, it demonstrated robust stability in complex tasks, ensuring semantic consistency [26, 27]. The architecture of Claude Sonnet 4 is independent of the models evaluated in this study (DeepSeek, GPT series, and Gemini), avoiding systematic evaluation bias caused by model homology. We generated adversarial inputs using the following strategies [11, 24, 28]:Semantic Equivalence Test: We replaced keywords in the question with common synonyms or more colloquial terms and reorganized the structure without changing the original meaning.Injecting Irrelevant Redundant Information: We added distracting information that was completely unrelated to the question, such as the patient’s hobbies and recent activities, while maintaining core information such as the disease name and key signs. Semantic Equivalence Test: We replaced keywords in the question with common synonyms or more colloquial terms and reorganized the structure without changing the original meaning. Semantic Equivalence Test: We replaced keywords in the question with common synonyms or more colloquial terms and reorganized the structure without changing the original meaning. Injecting Irrelevant Redundant Information: We added distracting information that was completely unrelated to the question, such as the patient’s hobbies and recent activities, while maintaining core information such as the disease name and key signs. Injecting Irrelevant Redundant Information: We added distracting information that was completely unrelated to the question, such as the patient’s hobbies and recent activities, while maintaining core information such as the disease name and key signs. We presented the rewritten questions to the five LLMs using the same standardized format mentioned above. Performance was then compared to that of the original questions to measure the decline in accuracy. The flowchart of this study is presented in Fig. 1.Fig. 1Flowchart of this study Flowchart of this study Statistical analysis All statistical analyses were conducted using RStudio version 2024.09.0 + 375 (Posit, Boston, MA, USA). Each model’s overall accuracy and accuracy across different subjects and question types were compared using chi-square tests and Fisher’s exact tests. Statistical significance was defined as p < 0.05. When statistical significance was detected, a post hoc analysis with Bonferroni correction was used for pairwise comparisons. McNemar’s test was employed to compare each model’s accuracy on the original versus the rewritten questions in the adversarial testing. Results Overall performance of LLMs As shown in Table 1, the overall accuracy differed significantly across the five LLMs (p < 0.001). Gemini 2.5 Pro achieved the highest accuracy at 91.67%, while GPT-4o had the lowest at 81.17%. Statistical analysis revealed significant differences between GPT-4o and the other models (p < 0.05), while no significant pairwise differences were detected among the other four LLMs (p > 0.05), as detailed in Table 2.Table 1Overall performance of five large language modelsCorrect%95% CIpDR154490.6788.05–92.87< 0.001*G4o48781.1777.80–84.22O353388.8386.04–91.24G5T54490.6788.05–92.87G2.555091.6789.16–93.75DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05Table 2Pairwise comparisons of LLMs’ for all questionsPairwise LLMsAll questionsDR1 vs G4o< 0.001*DR1 vs O31.000DR1 vs G5T1.000DR1 vs G2.51.000G4o vs O30.002*G4o vs G5T< 0.001*G4o vs G2.5< 0.001*O3 vs G5T1.000O3 vs G2.50.981G5T vs G2.51.000LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05 Table 1Overall performance of five large language modelsCorrect%95% CIpDR154490.6788.05–92.87< 0.001*G4o48781.1777.80–84.22O353388.8386.04–91.24G5T54490.6788.05–92.87G2.555091.6789.16–93.75DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05 Overall performance of five large language models DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro *p < 0.05 Table 2Pairwise comparisons of LLMs’ for all questionsPairwise LLMsAll questionsDR1 vs G4o< 0.001*DR1 vs O31.000DR1 vs G5T1.000DR1 vs G2.51.000G4o vs O30.002*G4o vs G5T< 0.001*G4o vs G2.5< 0.001*O3 vs G5T1.000O3 vs G2.50.981G5T vs G2.51.000LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05 Pairwise comparisons of LLMs’ for all questions LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro *p < 0.05 Performance of LLMs in five subjects Significant differences in accuracy were observed among the five models in both dentistry and clinical medicine (p < 0.05). However, in clinical medicine, the post hoc pairwise comparisons revealed no significant differences (p > 0.05). In dentistry, Gemini 2.5 Pro achieved the highest accuracy at 90.32%, while GPT-4o recorded the lowest at 79.35%. The accuracy of GPT-4o was significantly lower than the other four LLMs (p < 0.05). Detailed data are presented in Table S1 (see Additional file 1) and 3.Table 3Pairwise comparisons of LLMs’ for clinical medicine and dentistryPairwise LLMsClinical medicineDentistryDR1 vs G4o1.000< 0.001*DR1 vs O31.0001.000DR1 vs G5T1.0001.000DR1 vs G2.51.0001.000G4o vs O31.0000.002*G4o vs G5T1.000< 0.001*G4o vs G2.50.254< 0.001*O3 vs G5T1.0001.000O3 vs G2.50.5461.000G5T vs G2.51.0001.000LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05 Table 3Pairwise comparisons of LLMs’ for clinical medicine and dentistryPairwise LLMsClinical medicineDentistryDR1 vs G4o1.000< 0.001*DR1 vs O31.0001.000DR1 vs G5T1.0001.000DR1 vs G2.51.0001.000G4o vs O31.0000.002*G4o vs G5T1.000< 0.001*G4o vs G2.50.254< 0.001*O3 vs G5T1.0001.000O3 vs G2.50.5461.000G5T vs G2.51.0001.000LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05 Pairwise comparisons of LLMs’ for clinical medicine and dentistry LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro *p < 0.05 Performance of LLMs in ten sub-disciplines of dentistry Across the ten sub-disciplines of dentistry, significant performance differences were found among the five LLMs in prosthodontics and oral anatomy (p < 0.05). In prosthodontics, GPT-5 Thinking had the highest accuracy (85.71%), while GPT-4o had the lowest (65.18%). GPT-5 Thinking, DeepSeek-R1, and Gemini 2.5 Pro all outperformed GPT-4o (p < 0.05). In oral anatomy, Gemini 2.5 Pro achieved the highest accuracy at 100%, significantly exceeding GPT-4o (68.97%) (p = 0.019). Detailed data are presented in Table S2 (see Additional file 2) and 4.Table 4Pairwise comparisons of LLMs’ for prosthodontics and oral anatomyPairwise LLMsProsthodonticsOral anatomyDR1 vs G4o0.007*1.000DR1 vs O31.0001.000DR1 vs G5T1.0001.000DR1 vs G2.51.0000.518G4o vs O30.2581.000G4o vs G5T0.004*0.191G4o vs G2.50.023*0.019*O3 vs G5T1.0001.000O3 vs G2.51.0001.000G5T vs G2.51.0001.000LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05 Table 4Pairwise comparisons of LLMs’ for prosthodontics and oral anatomyPairwise LLMsProsthodonticsOral anatomyDR1 vs G4o0.007*1.000DR1 vs O31.0001.000DR1 vs G5T1.0001.000DR1 vs G2.51.0000.518G4o vs O30.2581.000G4o vs G5T0.004*0.191G4o vs G2.50.023*0.019*O3 vs G5T1.0001.000O3 vs G2.51.0001.000G5T vs G2.51.0001.000LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05 Pairwise comparisons of LLMs’ for prosthodontics and oral anatomy LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro *p < 0.05 Performance of LLMs in four question types As shown in Table S3 (see Additional file 3) and 5, significant differences in accuracy among the models were observed for both A1 and B1 questions (p < 0.05). For A1 questions, Gemini 2.5 Pro achieved the highest accuracy at 92.43%, while GPT-4o had the lowest (83.22%). Gemini 2.5 Pro and GPT-5 Thinking significantly outperformed GPT-4o (p < 0.05). For B1 questions, GPT-5 Thinking exhibited the highest accuracy (94.19%), while GPT-4o had the lowest (73.26%). The performance of GPT-4o was significantly worse than OpenAI o3, GPT-5 Thinking, and Gemini 2.5 Pro (p < 0.05).Table 5Pairwise comparisons of LLMs’ for A1 and B1 questionsPairwise LLMsA1B1DR1 vs G4o0.1220.061DR1 vs O31.0001.000DR1 vs G5T1.0001.000DR1 vs G2.51.0001.000G4o vs O30.0830.029*G4o vs G5T0.014*0.002*G4o vs G2.50.005*0.005*O3 vs G5T1.0001.000O3 vs G2.51.0001.000G5T vs G2.51.0001.000LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05 Table 5Pairwise comparisons of LLMs’ for A1 and B1 questionsPairwise LLMsA1B1DR1 vs G4o0.1220.061DR1 vs O31.0001.000DR1 vs G5T1.0001.000DR1 vs G2.51.0001.000G4o vs O30.0830.029*G4o vs G5T0.014*0.002*G4o vs G2.50.005*0.005*O3 vs G5T1.0001.000O3 vs G2.51.0001.000G5T vs G2.51.0001.000LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro*p < 0.05 Pairwise comparisons of LLMs’ for A1 and B1 questions LLMs Large language models, DR1 DeepSeek-R1, G4o GPT-4o, O3 OpenAI o3, G5T GPT-5 Thinking, G2.5 Gemini 2.5 Pro *p < 0.05 Performance of LLMs in adversarial testing Accuracy declined by 2.5%–5% in the semantic equivalence test. When irrelevant and redundant information was injected, the five LLMs showed an accuracy drop ranging from 1.66% to 5.84%. No significant accuracy drop was observed among the five LLMs under either perturbation relative to the original items (p > 0.05). Detailed results are presented in Table S4 (see Additional file 4). Discussion This research marks the first attempt to benchmark the performance of the new versions of LLMs in the context of the CDLE. Gemini 2.5 Pro, GPT-5 Thinking, DeepSeek-R1, and OpenAI o3 significantly outperform GPT-4o, and all models remained stable during adversarial testing. These findings highlight the potential for integrating these models into dentistry to improve dental education and practice. Regarding potential language bias, a prior study on the CDLE found that LLMs developed in China may hold a linguistic advantage when processing Chinese inputs [6]. Several cross-lingual studies have also suggested that the accuracy and logical consistency of LLMs such as ChatGPT might decline in non-English contexts [20, 23]. However, in our study, mainstream LLMs such as GPT-5 Thinking, Gemini 2.5 Pro, and OpenAI o3 performed comparably to DeepSeek-R1 in the context of the CDLE. This finding suggests that, as new-generation LLMs place greater emphasis on high-quality multilingual training data, optimize cross-lingual alignment, and improve tokenization, the disadvantages previously observed in non-native models due to linguistic bias and tokenization mechanisms may have been mitigated [4, 29, 30]. Given that DeepSeek-R1 not only showed performance similar to that of mainstream models, but is also specifically optimized for the Chinese context and offers the advantages of being open-source and free, it holds immense application potential in Chinese dental education and practice [1, 20]. This study included three representative models from OpenAI’s recent releases. The results demonstrate that OpenAI o3 outperformed GPT-4o in overall accuracy, while GPT-5 Thinking did not show a statistically significant improvement over OpenAI o3. This finding may reflect differences in model design as well as limitations of the evaluation benchmark. As a specialized reasoning model, OpenAI o3 is highly optimized for reasoning-intensive tasks [16], while GPT-5 Thinking, a deep-reasoning model within the GPT-5 system, is more likely to show advantages on open-ended, more complex tasks [18]. Furthermore, CDLE is a standardized examination, with well-structured questions and clearly defined medical concepts. Its difficulty level may fall within the capabilities of OpenAI o3, resulting in a performance ceiling that limits GPT-5 Thinking’s ability to demonstrate a clear advantage. Regarding performance across different specialties, the new versions of LLMs significantly outperformed GPT-4o in dentistry and its sub-disciplines (prosthodontics and oral anatomy), indicating substantial improvements in these areas. Recently, studies by Gokkurt Yilmaz et al. [7] and Yilmaz et al. [8] reported that ChatGPT o1 achieved accuracy over 96% on head and neck anatomy and oral pathology questions, and further pointed out significant performance differences among models, which echoes the findings of this study on the disciplinary dimension. However, when interpreting models’ performance across disciplines, uncertainty introduced by sample size must be taken into account. As shown in Tables S1 and S2 (see Additional file 1 and 2) , although all models achieved nearly 100% accuracy in preventive medicine (n = 9) and maxillofacial medical imaging diagnostics (n = 9) questions, the very small sample sizes led to wide 95% confidence intervals, making it difficult to rule out chance effects. Moreover, compared with other subjects, model accuracy in prosthodontics remained relatively low, consistent with prior research [5, 31]. This may be attributed to two factors. First, the number of questions varies by subject and may limit the ability to fully capture LLMs’ true proficiency [32]. Second, prosthodontics places higher demands on the model’s spatial thinking and logical reasoning [5]. Future research could explore targeted medical domain-specific fine-tuning and chain of retrieval approaches to further boost performance [33]. In terms of question types, the new versions of LLMs substantially outperformed GPT-4o on knowledge-based questions (A1 and B1). However, for case analysis questions (A2 and A3/A4), no significant differences were observed among the five models. This finding contrasts with the conclusions of Luo et al., who reported that GPT-4o showed significant improvements over GPT-4 and GPT-3.5 on case-based questions [32]. This indicates that the performance gains in current model iterations may be primarily concentrated in a more robust grasp of foundational concepts and factual knowledge, whereas improvements in clinical reasoning capabilities have not yet reached a comparable level. This may be attributable to two factors. First, LLM training relies heavily on text-based learning. This approach facilitates the acquisition of structured knowledge, enabling models to rapidly attain high proficiency in knowledge-based questions [34, 35]. Second, case analysis requires complex reasoning and rich clinical experience [20, 34]. It is difficult to achieve breakthroughs in this domain solely by expanding data scale. Consequently, in contrast to the rapid expansion of the knowledge base, improving logical reasoning capabilities remains considerably more challenging at the current stage. In adversarial testing, we applied two methods: semantic equivalence testing and the injection of irrelevant redundant information. These strategies were designed to simulate real-world scenarios characterized by unclear patient complaints, informational redundancy, and colloquial expressions likely to be used by medical students [24, 28]. We found that under these two perturbations, model accuracy declined only slightly, and changes were not statistically significant. This finding stands in contrast to research that explored the robustness of earlier-generation LLMs. Vishwanath et al. pointed out that distracting statements could reduce LLMs’ accuracy by up to 17.9% [24]. Hager et al. reported that LLMs’ accuracy declined by up to 18% when the presentation order of clinical information was changed [25]. These comparisons indicate that the robustness of newer LLM versions has improved substantially with model advances. This may be attributed to the incorporation of diverse perturbations and adversarial examples during their training, which in turn enhanced their ability to extract key information [36]. Unlike most previous evaluations that focused primarily on accuracy under standardized examination conditions [5, 12, 34], this study integrates accuracy benchmarking with adversarial testing, thereby extending the evaluation framework from simple accuracy to robustness in the face of input perturbations. This methodological refinement not only supplements the existing literature but also offers an assessment paradigm that more closely mirrors real-world scenarios. Notably, during adversarial testing, we only utilized one external model, Claude Sonnet 4, to generate perturbations. While this approach avoids self-generation and self-evaluation bias, the perturbation patterns are inevitably influenced by the linguistic style of a single external model, potentially introducing certain systematic biases. Future studies should combine manual curation with multiple external models to generate stronger perturbations that better mirror real-world scenarios, thereby further evaluating LLM robustness. The findings of this study highlight the robust performance of Gemini 2.5 Pro, GPT-5 Thinking, OpenAI o3, and DeepSeek-R1 on the CDLE, underscoring their substantial potential for integration into dental education and practice. In dental education, advanced LLMs could serve as personalized intelligent tutors that are tailored to each student’s interests and needs, strengthening their grasp of basic medical knowledge and skills [7, 8, 37]. Furthermore, LLMs could simulate clinical scenarios to help cultivate students’ clinical reasoning [8, 38]. In dental practice, for physicians, LLM-supported tools provide efficiency gains that may help mitigate burnout and greatly relieve heavy workloads [39–42]. For patients, especially those in remote areas lacking sufficient medical resources, LLMs may serve as consultants, analyzing dental problems and offering timely suggestions [43]. This study has several limitations. First, this assessment was restricted to structured MCQs. While the questions spanned multiple disciplines, they fall short of fully capturing the dynamic decision-making, patient-physician communication, and procedural contexts inherent in realistic settings. Therefore, caution is warranted when inferring the reasoning and interactive abilities of LLMs from these results. Second, because DeepSeek-R1 does not support multimodal input, we excluded all image-based questions. While this approach ensured a consistent evaluation baseline, it inevitably precluded the assessment of multimodal models’ proficiency in processing visual data. This constraint not only limited a comprehensive assessment of overall model capabilities but also weakened the generalizability of our findings to clinical scenarios that rely on rich visual data. Third, the adversarial testing employed only two types of perturbations, both generated by a single external model. Such a design cannot fully capture the diversity of linguistic styles and interference patterns encountered in real-world settings. Finally, given the rapid pace of LLM development, our evaluation reflects only the currently available model versions, which may limit generalizability to future iterations. Conclusions In conclusion, using the CDLE as a benchmark, new-generation LLMs achieved markedly higher accuracy. Furthermore, all models exhibited strong robustness against adversarial perturbations. These findings suggest that advanced LLMs hold promise as assistive tools for dental education and practice. However, prior to real-world deployment, comprehensive validation, strict regulation, and human supervision are mandatory. It should be noted that this study was limited to current model versions and structured MCQs, excluding multimodal data typical of real clinical scenarios. Future research should prioritize scenarios that align more closely with authentic clinical and educational practices, while continuously updating these findings as model versions evolve. Supplementary Information Additional file 1: Table S1 Performance of five large language models across subjects. Additional file 2: Table S2 presents the performance data of five large language models in ten sub-disciplines of dentistry. Additional file 3: Table S3 presents the performance data of five large language models in four question types. Additional file 4: Table S4 presents the performance data of five large language models in adversarial testing. Additional file 5: 600 multiple-choice questions and their answers. Additional file 6: Adversarial testing samples. Additional file 1: Table S1 Performance of five large language models across subjects. Additional file 2: Table S2 presents the performance data of five large language models in ten sub-disciplines of dentistry. Additional file 3: Table S3 presents the performance data of five large language models in four question types. Additional file 4: Table S4 presents the performance data of five large language models in adversarial testing. Additional file 5: 600 multiple-choice questions and their answers. Additional file 6: Adversarial testing samples.