Performance of large language models on the Turkish Pharmacy Specialty Examination: a comparative analysis of accuracy, confidence, and readability Authors: Ugur Balaban, Bahar Erbay, Mustafa Aktan, Kutay Demirkan Published: 2026 ABSTRACT Large language models (LLMs) have demonstrated strong performance in answering knowledge-based questions in healthcare education. Specialty examinations offer a standardized and objective framework to assess these capabilities. However, to date, no study has evaluated LLM performance on the Turkish Pharmacy Specialty Examination (EUS), a nationally standardized exam applied for the purpose of admitting candidates to pharmacy specialization programs. Therefore, this study aimed to comparatively evaluate LLM performance on EUS questions in terms of accuracy, self-reported confidence, and readability. This study conducted a comparative evaluation of three LLMs—ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash—using publicly available 84 multiple-choice questions from the EUS between 2017 and 2025. Each question was submitted to each model in a separate, newly initiated session using a standardized prompt. Model performance was assessed based on answer accuracy, self-reported confidence (1–5 scale), and readability of generated responses, using the Flesch reading ease (FRE), gunning fog index (GFI), and simple measure of Gobbledygook (SMOG) indices. All statistical analyses were performed using non-parametric repeated-measures methods, including Cochran’s Q test for paired categorical comparisons and the Friedman test with Durbin–Conover post-hoc analyses for readability scores, with two-tailed significance set at p < 0.05. Overall, the evaluated LLMs exhibited high performance. Gemini 2.5 Flash achieved the highest overall accuracy rate (92.9%), followed by ChatGPT-5.1 (90.5%) and DeepSeek-R1 (89.3%), with no statistically significant difference among the models (p = 0.584). Self-reported confidence was predominantly maximal (5/5), with ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash assigning maximum confidence to 87.5, 55.6, and 66.7% of incorrect responses, respectively. Significant differences in readability were observed among the evaluated LLMs. ChatGPT-5.1 generated texts with lower GFI and SMOG scores compared with DeepSeek-R1 and Gemini 2.5 Flash (p < 0.05), indicating lower linguistic complexity. No statistically significant differences were identified among models for FRE. LLMs demonstrated high and comparable accuracy when answering domain-specific pharmacy examination questions; however, occasional overconfidence in incorrect responses highlights the need for careful oversight. Differences in linguistic complexity underscore the importance of selecting models optimized for readability in educational settings. Overall, these findings suggest that LLMs may have potential as supplementary tools in pharmacy education within examination-based contexts, provided that expert guidance and critical appraisal are maintained to ensure reliability and clarity. Introduction Recent advances in natural language processing (NLP) have led to the rapid development of large language models (LLMs) that can understand, synthesize, and generate human-like text based on extensive datasets. These models have demonstrated significant potential in important healthcare-related tasks, including clinical documentation, patient communication, and clinical decision support1,2. LLMs can particularly support communication with patients by conveying medical information, providing translations and summaries, and improving patient care by facilitating information exchange. Furthermore, students can utilize LLMs to simulate taking medical histories from hypothetical patients, as well as applying diagnostic and treatment plans through simulation, thereby establishing LLMs as a significant tool in medical education2. In particular, widely used and continuously updated LLMs such as ChatGPT, DeepSeek and Gemini demonstrate high performance as various clinical decision support systems, offering promise in clinical decision-making processes1. Additionally, LLMs can assess medication regimens in the field of clinical pharmacotherapy to identify clinically significant drug-drug interactions, to determine clinical relevance levels, and to generate management recommendations3. Comprehensive evidence syntheses demonstrate that LLMs can generate clinically consistent responses in numerous healthcare fields. Systematic reviews and meta-analyses report that ChatGPT’s responses to knowledge-based medical questions in healthcare exhibit acceptable levels of accuracy and consistency, while emphasizing the necessity of verification prior to clinical application4,5. Specialty examinations have recently been adopted as objective tools for evaluating the performance of LLMs. Evidence from multiple-choice examinations across diverse medical disciplines indicates that LLMs can achieve good overall accuracy, surpassing conventional passing thresholds6–9. Despite the growing body of evidence in medicine and healthcare, the role of LLMs in pharmacy education and the pharmacy specialty examination has largely been unexplored in Türkiye. The Turkish Pharmacy Specialty Examination (EUS) is a nationally standardized assessment exam, required for admission to clinical pharmacy and phytopharmacy specialization programmes following completion of a pharmacy degree. It is administered by the Student Selection and Placement Centre (ÖSYM) (https://www.osym.gov.tr), a governmental institution responsible for assessing and placing qualified applicants into higher education programmes through central examinations. The examination is designed in accordance with the national core pharmacy education curriculum and evaluates comprehensive domain-specific knowledge within a limited time frame. As such, the EUS provides a robust and discipline-specific framework for assessing the capabilities of LLMs in pharmacy education. To date, no studies have specifically evaluated LLM performance in the EUS. Therefore, this study aims to conduct a comparative analysis of ChatGPT, DeepSeek, and Gemini using EUS questions, focusing on accuracy, response confidence, and readability. Materials and methods Multiple-choice questions (MCQs), sourced from EUS examinations applied by ÖSYM between 2017 and 2025, were analyzed by three different LLMs. To ensure compliance with copyright and ethical standards, examination questions only announced to the public (10% of the total questions) were included in the analysis. Categorization of exam questions The EUS consisted of 100 questions from 2017 to 2022 and was revised to 75 questions for the period 2023–2025. Exam questions were categorized into two main domains: “Pharmaceutical Basic Sciences” and “Pharmaceutical Professional Sciences”. Pharmaceutical Basic Sciences: Analytical chemistry, physiology-pathology, biochemistry, pharmaceutical microbiology and immunology, medical biology. Pharmaceutical Professional Sciences: Pharmacy legislation/ethics, pharmacology, clinical pharmacy, pharmaceutical chemistry, pharmacognosy/pharmaceutical botany, pharmaceutical technology, pharmaceutical toxicology. Model versions and access dates The study was conducted between 1st and 10th of December 2025. During this period, the publicly available versions of each model were accessed via the official web interface. Details of the models are as follows: ChatGPT-5.1 (OpenAI, GPT-5, 2025 version) accessed via web interface (https://chat.openai.com) between 1 and 10 December 2025. Gemini 2.5 Flash (Google DeepMind, Gemini Flash 2.5) accessed via web interface (https://gemini.google.com) between 1 and 10 December 2025. DeepSeek-R1 (DeepSeek AI, 2025 version) accessed via web interface (https://chat.deepseek.com) between 1 and 10 December 2025. Prompting protocol All questions were presented in their original Turkish multiple-choice format, each consisting of five options (A–E). No additional hints, explanations, or contextual information were provided. All models were run using the default parameters available on their respective platforms. For each model, a new chat session was initiated for every question to eliminate potential carryover effects from previous interactions. All models were instructed to respond in Turkish. A standardized prompt was used consistently across all models, as follows: “Can you answer this multiple-choice question? Also, on a scale of one to five, indicate how confident you felt when answering this question.” Evaluation of responses The performance of the model was evaluated by using three primary metrics: accuracy, self-confidence, and text readability. Accuracy and confidence Accuracy was defined as the exact match between the option selected by the model (A–E) and the predefined answer key. Since each question had only one correct answer, the accuracy assessment was objective. The model’s self-confidence was evaluated by requesting it to rate its confidence level for each response on a scale from 1 to 5, where higher scores indicated greater confidence. Readability and text complexity Readability and text complexity were assessed using three established readability indices: Flesch Reading Ease (FRE), Gunning Fog Index (GFI), and SMOG (Simple Measure of Gobbledygook). To assess linguistic complexity, the full responses—including detailed clinical explanations and reasoning—generated by the LLMs for every individual question were analyzed using an online readability scoring system. Because the evaluated readability indices were originally developed for English-language texts, all Turkish LLM-generated responses were translated into English prior to analysis in order to standardize linguistic assessment across models. Translation was performed uniformly using the same machine-translation workflow for all responses to minimize systematic language-related variability. The translated texts were subsequently analyzed using the same readability scoring platform. All readability scores were calculated using the Readability Scoring System available at https://readabilityformulas.com/readability-scoring-system.php. The FRE score ranges from 0 to 100, with higher scores indicating easier readability. It is calculated based on the average number of syllables per word and the average number of words per sentence. Scores between 0 and 30 represent very confusing text, 30–50 indicate difficult readability, 50–60 moderately difficult readability, 70–80 fairly easy readability, and 80–100 very easy readability10. The GFI estimates the years of formal education required to understand the text. It is calculated based on the average sentence length and the proportion of complex words within the text. Higher GFI scores indicate greater text complexity and a higher educational level needed for comprehension11. The SMOG index is another widely used readability measure that estimates the number of years of education required to understand a text on first reading. It is based on the number of polysyllabic words (words with three or more syllables) and the number of sentences in the text. Higher SMOG scores reflect more complex and less readable text, whereas lower scores indicate simpler and more accessible content12. Statistical analysis Statistical analyses were performed using R software (R version 4.5.2; R Foundation for Statistical Computing, Vienna, Austria). Normality of the readability scores (FRE, GFI, SMOG) was assessed using the Shapiro–Wilk test. As the data did not meet the assumptions of normal distribution, non-parametric tests were applied. Descriptive statistics are presented as counts and percentages for accuracy data, and medians with interquartile ranges (IQR) for readability scores. Since each of the 84 examination questions was independently evaluated by all three LLMs, the resulting observations represented paired repeated-measures data. Therefore, differences in binary accuracy outcomes among the three models were assessed using Cochran’s Q test. Readability scores across the three LLMs were compared using the Friedman test for repeated non-parametric measures, followed by Durbin–Conover post-hoc pairwise comparisons with adjusted p-values for multiple testing. All analyses were two-tailed, and statistical significance was set at p < 0.05. Results Accuracy rates of LLMs A total of 84 questions were included in the analysis, comprising 23 (27.4%) from pharmaceutical basic sciences and 61 (72.6%) from pharmaceutical professional sciences. Across both domains, all three LLMs demonstrated high accuracy (Table 1; Fig. 1). When all 84 questions were analyzed together, overall accuracy varied between 89.3% and 92.9%, with no statistically significant difference between the three models (p = 0.584). In pharmaceutical basic sciences, ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash achieved accuracy values ranging from 87% to 100%, with no statistically significant difference among the models (p = 0.097). Similarly, in pharmaceutical professional sciences, accuracy ranged from 88.5% to 90.2%, with no statistically significant difference in performance (p = 0.905). Table 1Comparative accuracy of LLMs across pharmaceutical science categories and overall performance.CategoryModelCorrect (n)Incorrect (n)Accuracy (%)p-valuePharmaceutical basic sciences, (n = 23)ChatGPT22195.70.097DeepSeek20387.0Gemini230100Pharmaceutical professional sciences, (n = 61)ChatGPT54788.50.905DeepSeek55690.2Gemini55690.2Total, (n = 84)ChatGPT76890.50.584DeepSeek75989.3Gemini78692.9LLM, large language model. Table 1Comparative accuracy of LLMs across pharmaceutical science categories and overall performance.CategoryModelCorrect (n)Incorrect (n)Accuracy (%)p-valuePharmaceutical basic sciences, (n = 23)ChatGPT22195.70.097DeepSeek20387.0Gemini230100Pharmaceutical professional sciences, (n = 61)ChatGPT54788.50.905DeepSeek55690.2Gemini55690.2Total, (n = 84)ChatGPT76890.50.584DeepSeek75989.3Gemini78692.9LLM, large language model. Comparative accuracy of LLMs across pharmaceutical science categories and overall performance. LLM, large language model. Fig. 1Accuracy rates of ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash across pharmaceutical basic and professional sciences. Accuracy rates of ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash across pharmaceutical basic and professional sciences. Across specific pharmaceutical basic and professional domains, the evaluated LLMs demonstrated high and consistent accuracy (Table 2). In the basic sciences, performance ranged from 66.7% to 100%. All models achieved 100% accuracy in physiology–pathology, biochemistry and medical biology, and pharmaceutical microbiology and immunology, while analytical chemistry showed lower performance for ChatGPT-5.1 (88.9%) and DeepSeek-R1 (66.7%). In the professional sciences, accuracy remained high in pharmacology and clinical pharmacy (92.3%–100%), pharmaceutical chemistry (83.3%–91.7%), pharmacognosy/pharmaceutical botany (92.3%–100%), pharmaceutical technology (83.3%–100%), and pharmaceutical toxicology (100%). By contrast, pharmacy legislation and ethics showed the lowest performance, with ChatGPT-5.1 and Gemini 2.5 Flash at 60% and DeepSeek at 20%. Table 2Subject area-specific accuracy rates of LLMs in pharmaceutical sciences.CategoryAccuracy ratesChatGPT n (%) DeepSeek n (%) Gemini n (%) Pharmaceutical basic sciencesAnalytical chemistry, (n = 9)8 (88.9)6 (66.7)9 (100)Physiology-pathology, (n = 5)5 (100)5 (100)5 (100)Biochemistry and medical biology, (n = 5)5 (100)5 (100)5 (100)Pharmaceutical microbiology and immunology, (n = 4)4 (100)4 (100)4 (100)Pharmaceutical professional sciencesPharmacy legislation/ethics, (n = 5)3 (60)1 (20)3 (60)Pharmacology and clinical pharmacy, (n = 13)13 (100)13 (100)12 (92.3)Pharmaceutical chemistry, (n = 12)10 (83.3)10 (83.3)11 (91.7)Pharmacognosy/pharmaceutical botany, (n = 12)11 (91.7)12 (100)11 (91.7)Pharmaceutical technology, (n = 12)10 (83.3)12 (100)11 (91.7)Pharmaceutical toxicology, (n = 7)7 (100)7 (100)7 (100)LLM, large language models. Table 2Subject area-specific accuracy rates of LLMs in pharmaceutical sciences.CategoryAccuracy ratesChatGPT n (%) DeepSeek n (%) Gemini n (%) Pharmaceutical basic sciencesAnalytical chemistry, (n = 9)8 (88.9)6 (66.7)9 (100)Physiology-pathology, (n = 5)5 (100)5 (100)5 (100)Biochemistry and medical biology, (n = 5)5 (100)5 (100)5 (100)Pharmaceutical microbiology and immunology, (n = 4)4 (100)4 (100)4 (100)Pharmaceutical professional sciencesPharmacy legislation/ethics, (n = 5)3 (60)1 (20)3 (60)Pharmacology and clinical pharmacy, (n = 13)13 (100)13 (100)12 (92.3)Pharmaceutical chemistry, (n = 12)10 (83.3)10 (83.3)11 (91.7)Pharmacognosy/pharmaceutical botany, (n = 12)11 (91.7)12 (100)11 (91.7)Pharmaceutical technology, (n = 12)10 (83.3)12 (100)11 (91.7)Pharmaceutical toxicology, (n = 7)7 (100)7 (100)7 (100)LLM, large language models. Subject area-specific accuracy rates of LLMs in pharmaceutical sciences. LLM, large language models. Self-reported confidence ratings of LLMs When asked to rate their confidence on a 1-to-5 scale, all three models reported maximum confidence level of 5 for most responses. The only deviation from maximum confidence was a slightly lower level of 4; no responses received confidence levels of 1, 2, or 3. ChatGPT-5.1 assigned a slightly lower confidence level of 4 to one question in pharmaceutical technology, DeepSeek-R1 to four questions (analytical chemistry, n = 2; pharmacy legislation/ethics, n = 1; pharmaceutical chemistry, n = 1), and Gemini 2.5 Flash to three questions (pharmaceutical chemistry, pharmacognosy/pharmaceutical botany, and pharmacology and clinical pharmacy). All remaining questions were rated as 5 by all models. Across the 84 questions, ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash produced 8 (9.5%), 9 (10.7%), and 6 (7.1%) incorrect responses, respectively. As shown in Fig. 2, most incorrect responses were assigned the maximum confidence level of 5 across all models. ChatGPT-5.1 assigned maximum confidence to 7 out of 8 incorrect responses (87.5%), while DeepSeek-R1 assigned maximum confidence to 5 out of 9 incorrect responses (55.6%). Gemini 2.5 Flash assigned maximum confidence to 4 out of 6 incorrect responses (66.7%). Fig. 2Self-reported confidence ratings for incorrect responses by ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash. Self-reported confidence ratings for incorrect responses by ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash. Readability assessment of LLM-generated texts Readability scores for texts generated by ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash are presented in Table 3. All LLM-generated texts exhibited low FRE scores, indicating substantial reading difficulty, with no statistically significant differences between models (p = 0.790). In contrast, both the GFI (p = 0.001) and SMOG (p < 0.001) differed significantly across LLMs, with ChatGPT-5.1 producing relatively more readable texts, whereas DeepSeek-R1 and Gemini 2.5 Flash generated more complex texts requiring higher educational attainment for comprehension. Table 3Readability indexes of texts generated by LLMs.Readability indexesChatGPTDeepSeekGeminip-valueFRE, median (IQR)31 (33.75)29 (24)28 (22)0.790GFI, median (IQR)13.95 (4.78)14.6 (5.02)15.35 (3.18) 0.001 SMOG, median (IQR)9.19 (2.55)11.79 (3.58)11.31 (2.73) < 0.001 FRE, Flesch Reading Ease; GFI, Gunning Fog Index; IQR, interquartile range; LLM, large language model; SMOG, Simple Measure of Gobbledygook. Bold values indicate statistical significance at p < 0.05. Table 3Readability indexes of texts generated by LLMs.Readability indexesChatGPTDeepSeekGeminip-valueFRE, median (IQR)31 (33.75)29 (24)28 (22)0.790GFI, median (IQR)13.95 (4.78)14.6 (5.02)15.35 (3.18) 0.001 SMOG, median (IQR)9.19 (2.55)11.79 (3.58)11.31 (2.73) < 0.001 FRE, Flesch Reading Ease; GFI, Gunning Fog Index; IQR, interquartile range; LLM, large language model; SMOG, Simple Measure of Gobbledygook. Bold values indicate statistical significance at p < 0.05. Readability indexes of texts generated by LLMs. FRE, Flesch Reading Ease; GFI, Gunning Fog Index; IQR, interquartile range; LLM, large language model; SMOG, Simple Measure of Gobbledygook. Bold values indicate statistical significance at p < 0.05. Post-hoc pairwise comparisons showed that ChatGPT-5.1 generated significantly lower GFI scores than DeepSeek-R1 (p = 0.017) and Gemini 2.5 Flash (p < 0.001). Similarly, ChatGPT produced significantly lower SMOG scores compared with both Gemini 2.5 Flash and DeepSeek-R1 (p < 0.001 for both), indicating lower linguistic complexity and greater readability. No statistically significant differences were observed between DeepSeek-R1 and Gemini 2.5 Flash for either GFI or SMOG, and no pairwise differences were identified for FRE (Table 4). Table 4Pairwise post-hoc comparisons of readability indexes between LLMs.Readability indexesLLM-1LLM-2p-valueFREChatGPTGemini0.532ChatGPTDeepSeek0.938GeminiDeepSeek0.584GFIChatGPTGemini < 0.001 ChatGPTDeepSeek 0.017 GeminiDeepSeek0.188SMOGChatGPTGemini < 0.001 ChatGPTDeepSeek < 0.001 GeminiDeepSeek0.306FRE, Flesch Reading Ease; GFI, Gunning Fog Index; IQR, interquartile range; LLM, large language model; SMOG, Simple Measure of Gobbledygook. Bold values indicate statistical significance at p < 0.05. Table 4Pairwise post-hoc comparisons of readability indexes between LLMs.Readability indexesLLM-1LLM-2p-valueFREChatGPTGemini0.532ChatGPTDeepSeek0.938GeminiDeepSeek0.584GFIChatGPTGemini < 0.001 ChatGPTDeepSeek 0.017 GeminiDeepSeek0.188SMOGChatGPTGemini < 0.001 ChatGPTDeepSeek < 0.001 GeminiDeepSeek0.306FRE, Flesch Reading Ease; GFI, Gunning Fog Index; IQR, interquartile range; LLM, large language model; SMOG, Simple Measure of Gobbledygook. Bold values indicate statistical significance at p < 0.05. Pairwise post-hoc comparisons of readability indexes between LLMs. FRE, Flesch Reading Ease; GFI, Gunning Fog Index; IQR, interquartile range; LLM, large language model; SMOG, Simple Measure of Gobbledygook. Bold values indicate statistical significance at p < 0.05. Discussion This study evaluated the performance of three LLMs across accuracy, self-reported confidence, and readability using a standardized set of pharmacy specialization examination questions. Overall, all models demonstrated high and comparable accuracy across both pharmaceutical basic and professional sciences, with no statistically significant differences observed. Confidence ratings were predominantly at the maximum level. Readability analyses, however, revealed significant differences in linguistic complexity, with ChatGPT-5.1 generating more readable texts than Gemini 2.5 Flash and DeepSeek-R1. These results indicate that while current LLMs perform well in answering domain-specific examination questions, differences remain in how clearly responses are communicated and in the alignment between confidence and correctness, which may have implications for their use in educational settings. In the present study, Gemini 2.5 Flash achieved the highest overall accuracy rate, and all LLMs demonstrated high performance. Our results are in line with previous research reporting strong LLM performance in domain-specific examinations9,13–15. A recent study on the Turkish Dental Specialty Examination (DUS) evaluated ChatGPT-4o and Gemini Advanced against the top-performing candidates using publicly available examination questions. Both LLMs were able to pass the DUS by exceeding the minimum threshold score8. Similarly, Koçak et al.16 assessed ChatGPT-4, Gemini 1.5 Pro, Command R+, and LLaMA-3 70B on 240 questions from the Turkish Medical Specialty Training Entrance Exam (TUS), reporting that ChatGPT-4 achieved the highest overall accuracy and that ChatGPT-4, LLaMA-3 70B, and Gemini 1.5 Pro all significantly outperformed the average human performance. In clinically oriented tasks, ChatGPT-4, GPT-3.5, Claude-2, and LLaMA2-7B demonstrated strong decision-making using patient cases prepared by critical care pharmacists, yet caution is warranted when using LLMs for clinical decision support, particularly in complex medication management17. Another study evaluating the performance of 14 LLMs in antibiotic prescribing demonstrated that, although ChatGPT-o1 performed with the highest accuracy, substantial variability across models and inconsistent results in prescribing appropriate antibiotic selection, dosing, and treatment durations highlight their limited reliability for clinical decision-making18. These findings suggest that while LLMs show remarkable proficiency in domain-specific examinations and educational tasks, their application in complex clinical decision-making remains limited and should be approached with caution. Regarding readability assessment, no statistically significant difference was observed in FRE scores between the models. However, ChatGPT-5.1 demonstrated significantly lower GFI and SMOG scores compared to the other models, indicating higher readability than DeepSeek-R1 and Gemini 2.5 Flash. Similarly, a study assessing the quality of bladder cancer information generated from various LLMs (ChatGPT-4o, Google Gemini 2.0 Flash, Grok 3, Claude 3.7 Sonnet, and DeepSeek R1) found that ChatGPT-4o had the highest FRE score with an average of 59.4 and the lowest SMOG score with an average of 7.88 compared to other LLMs19. In contrast, a separate assessment using FRES and SMOG indices reported that although DeepSeek-R1 demonstrated significantly higher readability with higher FRES and lower SMOG scores compared to ChatGPT-4o, both models were considered suitable for clinical pharmacy education20. Furthermore, an evaluation of accuracy, completeness, and readability regarding Familial Mediterranean Fever-associated anemia across ChatGPT, Copilot, and Gemini revealed no statistically significant differences among the models21. Collectively, these findings suggest that while subtle differences exist between LLM versions and specific medical topics, most models provide sufficient readability for educational and clinical purposes, though careful selection of the model may optimize comprehension and usability. LLMs have recently become among the most frequently used tools for answering medical examinations across various fields. Although artificial intelligence-supported chatbots have demonstrated superior performance in studies conducted in medicine and pharmacy, they remain professionally insufficient in certain aspects22. Integrating current LLMs into medical and pharmacy education may facilitate rapid and easy access to information; however, it should not be overlooked that, in some cases, they may lead to misinformation due to the “hallucination phenomenon”23,24. Our findings indicated a misalignment between confidence levels and accuracy, with both ChatGPT-5.1, DeepSeek-R1 and Gemini 2.5 Flash frequently assigning maximum confidence to incorrect answers. Such patterns suggest that reliance on these models in pharmaceutical practice may require careful oversight to maintain the necessary standards of factual accuracy. It is important to note that self-reported confidence in LLMs does not necessarily correspond to calibrated probabilistic estimates of correctness and may not reliably reflect actual model performance25,26. Therefore, these confidence ratings should not be interpreted as true probabilities of correctness in clinical or educational decision-making contexts. This study has several limitations. First, only the publicly announced portion of the EUS questions was used, which prevents a direct comparison with the full knowledge level of human candidates. Second, because the evaluated EUS questions were publicly accessible online before this study was conducted, it cannot be excluded that similar or identical content may have been encountered during model pretraining or indexing processes. Given the limited transparency regarding proprietary training datasets and model cutoff dates, the extent of prior exposure could not be determined. Therefore, the observed performance may partially reflect memorization or familiarity with publicly available examination materials rather than entirely de novo reasoning capability. Third, the sample size of questions was limited, which may affect the precision and generalizability of the findings. Fourth, the study was conducted in Turkish and within the specific context of the EUS exam, limiting the applicability of results to other languages or educational systems. Additionally, readability formulas such as FRE, GFI, and SMOG are primarily designed for general English texts and may have limited applicability to technical, domain-specific medical content, where specialized terminology may influence the resulting scores. Finally, although the models reported high confidence, incorrect answers occurred, highlighting a discrepancy between self-reported certainty and actual accuracy. Despite these limitations, the study offers valuable insights into the comparative performance of LLMs on the EUS, delineating their strengths and weaknesses in handling domain-specific questions and providing a foundation for future investigations in the evaluation of LLMs in pharmacy examinations. Conclusion The evaluated LLMs demonstrated promising performance in responding to pharmacy specialization examination questions, with ChatGPT-5.1 generating responses with lower linguistic complexity according to GFI and SMOG metrics. However, the occurrence of incorrect answers despite maximum self-reported confidence highlights a critical risk of overconfident misinformation. These findings suggest that while LLMs may serve as valuable supplementary tools in pharmacy education, they should be approached with caution and not be utilized as standalone resources. Expert supervision, critical appraisal skills, and cross-verification using multiple models remain essential to ensure the safe and reliable integration of LLMs into educational practice.