Assessing diagnostic performance of multimodal LLMs and a custom convolutional neural network in tooth-level caries detection and localization Authors: Aamna Khalid, Ayesha Nooruddin, Nighat Naved, Syed Muhammad Faizan Ahmed, Niha Adnan, Sarang Suresh, Abhishek Lal, Fahad Umer Published: 2026 ABSTRACT BackgroundArtificial Intelligence is reshaping dental diagnostics through automated interpretation of images. While Convolutional Neural Networks (CNNs) demonstrate high accuracy via domain-specific training, multimodal Large Language Models (LLMs) such as ChatGPT-4o and Gemini 2.5 Flash have recently acquired visual-reasoning capabilities without task-specific fine-tuning.ObjectiveThis study compared the diagnostic performance of these LLMs with a custom CNN for detecting and localizing dental caries on intraoral images.MethodsThis cross-sectional diagnostic accuracy study used 22 occlusal-view intraoral images. ChatGPT-4o, Gemini 2.5 Flash, and a YOLOv5s-based CNN analyzed each image for caries detection and localization. Quantitative evaluation assessed decay detection using accuracy, sensitivity, specificity, precision, Positive Predictive Value (PPV), Negative Predictive Value (NPV), and F1 score. Inter-model differences were analyzed using McNemar’s test. Additionally, a descriptive qualitative evaluation was performed by specialist dentists, who rated each model’s output for realism, diagnostic accuracy, bounding-box precision, and absence of unnecessary annotations using a 3-point Likert scale.ResultsThe CNN achieved the highest diagnostic accuracy (97.2%), sensitivity (86.7%), and F-1 score (88.0%). Gemini 2.5 Flash outperformed ChatGPT-4o in sensitivity (76.4 vs. 66.2%) and F-1 score (74.3 vs. 68.7%). Overall, CNN’s performance was significantly superior (p < 0.001), whereas no difference was found between the two LLMs (p = 0.541). Qualitatively, CNN scored best for realism (90.9%), decay accuracy (79.5%), and bounding-box precision (93.1%).ConclusionCNNs provide superior accuracy for caries localization compared with multimodal LLMs. However, LLMs demonstrate potential for generating clinically interpretable diagnostic summaries. Hybrid systems integrating CNN-based detection with LLM-driven reasoning may enhance decision-making and improve efficiency in dental diagnostic workflows.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12903-026-08590-2. Introduction Artificial Intelligence (AI) is rapidly transforming dental practice by introducing smarter, faster, and more accurate ways to diagnose and manage oral health conditions [1]. Through the use of Machine Learning (ML) algorithms, AI can analyze intraoral images, radiographs and patient records with remarkable precision [2]. This shift does not replace dentists; rather, it supports them by making clinical decisions more data-driven, improving patient outcomes, and saving valuable time [3]. As AI tools are increasingly being integrated into daily dental practice, they are setting new standards for efficiency, consistency, and quality in dental care [4]. Building on this progress, ML and Deep Learning (DL) algorithms, particularly Convolutional Neural Networks (CNNs) have demonstrated noteworthy performance in dental diagnostics [5]. Several systematic reviews have reported high accuracies, reaching up to 99.1% for detecting dental caries using smartphones-based applications [6, 7]. This highlights the growing reliability and clinical utility of CNN models. However, training such models to perform domain-specific task is quite resource-intensive, requiring high quality expert-annotated training datasets and substantial computational power [8]. One of the most notable advancements is the development of Large Language Models (LLMs), like ChatGPT, Llama, and Gemini [9]. Initially built for Natural Language Processing (NLP) tasks, their newer versions such as ChatGPT-5, ChatGPT-4o and Gemini 2.5 Flash now support multimodal capabilities, enabling them to interpret various inputs including text, audio and images [10]. With these advancements, LLMs are increasingly being explored in healthcare for tasks like summarizing patient records, answering clinical queries, and simulating interactions between doctors and patients [11]. With their user-friendly interface and wide accessibility, LLMs have become increasingly accessible for everyday users including clinicians seeking a quick second opinion as well as patients [12]. Many patients, in particular, feel comfortable using them because they provide quick and easy-to-understand responses in a conversational manner [13]. This familiarity has made LLMs appealing tools for people seeking clarification about their medical or dental conditions [14]. Nonetheless, the question arises, can general-purpose LLMs, without any domain-specific training, match or surpass the diagnostic performance of specialized CNNs? Numerous studies have validated the use of CNN-based models for dental diagnostics using both textual and image data [15–17]. However, to the best of the authors’ knowledge, no study has directly compared the performance of a specialized CNN model with widely used LLMs such as ChatGPT-4o and Gemini 2.5 Flash for analyzing clinical intraoral images. Therefore, this study seeks to address this gap by evaluating the performance of ChatGPT-4o and Gemini 2.5 Flash with a specialized CNN in performing domain-specific clinical tasks, such as caries detection and localization on intraoral images, by comparing them with a human Reference Standard (RS). Research question What is the diagnostic performance of LLMs (ChatGPT-4o and Gemini 2.5 Flash) and a specialized, custom CNN model against an expert-generated RS in detecting and localizing dental caries on intraoral images? Materials and methods Ethical approval The study was conducted in accordance with the ethical guidelines set by the Ethical Review Committee (ERC) of the institution, Aga Khan University Hospital (AKUH). (ERC# 2024-9433-32047). Written informed consent and assent were obtained from all participants, allowing their anonymized data to be used for research purposes. Sample size calculation Sample size was estimated using WHO sample size calculator. Based on a previously reported study the difference in diagnostic performance (precision) of LLMs versus CNN model for detection of oral pathology lesion was found to be 29.39 [18]. With a Confidence Interval (CI) of 95% and power of the study at 80, the sample size was calculated to be 22 images. The same set of 22 images were used for each model. Since the unit of analysis was tooth and each image had approximately 14 teeth therefore the total sample size turned out to be 308 (22*14). Study design Cross-sectional, diagnostic accuracy study. Image testing dataset A total of 22 anonymized intraoral images (maxillary and mandibular occlusal views) in JPEG format were randomly selected from the dataset utilized in a previous study [19]. This dataset was utilized because it was collected using a standardized protocol, ensuring consistent image quality and annotation. Moreover, it originates from a region with a high prevalence of dental caries, making it particularly relevant for the evaluation of such models. On average, each image had 14 teeth visible, and each tooth was treated as a separate entity for statistical analysis. Images that were excessively blurry, obstructed by saliva or blood, or had significant lighting artifacts were excluded to prevent misinterpretation. Model description ChatGPT-4o model The ChatGPT-4o model, equipped with multimodal input capabilities, was utilized to analyze the testing dataset. Each intraoral image was submitted using a standardized prompt instructing the model to identify dental caries and generate bounding boxes around the affected teeth. The model was accessed via OpenAI’s free-tier web interface under default platform conditions. Inference hyperparameters such as temperature, top-p, and random seed were not user-configurable in this interface and therefore could not be specified. All outputs were generated using the platform’s default inference configuration at the time of testing, reflecting typical user conditions. Gemini 2.5 Flash The Gemini 2.5 Flash multimodal model, capable of processing and analyzing both text and visual inputs, was used to evaluate the same set of intraoral images. The analysis was conducted using model’s default settings interface. Inference hyperparameters such as temperature, top-p, and random seed were not exposed to users in this interface and therefore could not be modified. CNN model A CNN architecture, based on YOLOv5s was trained for object detection to localize carious lesions. The model was developed using a dataset comprising 13,561 training images, 1,695 validation images, and 1,695 testing images, collected as part of authors' another study conducted in Mithi, Tharparkar, Pakistan. Two general dentists (A.K, A.N), each with over two years of clinical and annotation experience, independently annotated visible caries using bounding boxes on the LabelMe annotation tool (Version 5.4.1) [20] Inter-annotator agreement was near-perfect (Cohen’s kappa = 0.89). Disagreements were reviewed and resolved by a specialist restorative dentist with over five years of experience, ensuring a high-quality ground truth. The model generated bounding box coordinates around decayed teeth, thereby identifying both the presence and location of dental caries. It was trained for 300 epochs using standard hyperparameters, early stopping (patience = 100), and the Adam optimizer on an NVIDIA RTX 4090 (24 GB GPU) with a batch size of 54. This was implemented using the PyTorch framework to ensure efficient training and deployment. Prompting strategy Response to different prompts To assess how prompt structure influenced the diagnostic performance of ChatGPT-4o and Gemini 2.5 Flash, different types of prompts were tested by two general dentists (A.K, A.N). These prompts were further verified by a specialist endodontist (N.N). The goal was to determine whether a more detailed and context-rich prompt would enhance the accuracy of caries detection and localization. Based on comparative results, the following prompt was selected for final analysis. All previously tested prompts are provided in Supplementary file 1. No prompt was required for the CNN model, as it operates solely by processing uploaded images without textual input. Final prompt The following prompt was provided to both ChatGPT-4o and Gemini 2.5 Flash, without any model-specific optimization. "You are a dental expert specializing in the detection of dental caries. I will provide you with a high-resolution intraoral image that shows the occlusal view. Your task is to carefully analyze the image and detect all visible dental decay, specifically, carious lesions on molars, premolars, or any other teeth that exhibit clear signs of cavitation or dark discoloration indicative of decay. Please mark each decayed area with a distinct bounding box, ensuring that the boxes accurately enclose only the affected regions. Do not include healthy teeth or areas with superficial staining that do not suggest caries. Once you have identified the decayed teeth, return the same image with the bounding boxes clearly overlaid. Focus on visual accuracy and exclude any explanatory text, only return the annotated image. Also list the affected teeth using FDI notation, along with a brief description of each lesion’s location and apparent severity." "You are a dental expert specializing in the detection of dental caries. I will provide you with a high-resolution intraoral image that shows the occlusal view. Your task is to carefully analyze the image and detect all visible dental decay, specifically, carious lesions on molars, premolars, or any other teeth that exhibit clear signs of cavitation or dark discoloration indicative of decay. Please mark each decayed area with a distinct bounding box, ensuring that the boxes accurately enclose only the affected regions. Do not include healthy teeth or areas with superficial staining that do not suggest caries. Once you have identified the decayed teeth, return the same image with the bounding boxes clearly overlaid. Focus on visual accuracy and exclude any explanatory text, only return the annotated image. Also list the affected teeth using FDI notation, along with a brief description of each lesion’s location and apparent severity." Platform constraints ChatGPT-4o was accessed through the free-tier version of the platform, which imposed specific usage limitations. A maximum of two images could be submitted concurrently, and a daily cap on image uploads was enforced. Upon reaching this limit, data collection was paused for 12 h before resuming. In contrast, Gemini 2.5 Flash did not impose any prompt or upload restrictions, therefore, all images were processed in a single session. Response validation and inclusion criteria All outputs were independently reviewed (A.K, A.N), following the STARD-AI checklist, provided in Supplementary file 2 [21]. Prompts were repeated up to two times only when the generated outputs were completely irrelevant, such as when the image was altered or displayed additional, non-existent teeth. Only the outputs containing bounding boxes were included in the final evaluation, even if some boxes were inaccurately placed or included non-carious areas. Types of responses from models All three models produced outputs in the form of annotated images, with bounding boxes drawn around areas they identified as carious. In addition, they provided descriptive findings in the form of summary tables as shown in Fig. 1. Fig. 1Illustrates the input, output and evaluation of all three models. The textual responses were compared with RS, who were unaware of the model’s results and the image generated by the models were evaluated by specialist endodontists. CNN: Convolutional Neural Network, RS: Reference Standard Illustrates the input, output and evaluation of all three models. The textual responses were compared with RS, who were unaware of the model’s results and the image generated by the models were evaluated by specialist endodontists. CNN: Convolutional Neural Network, RS: Reference Standard Establishment of Reference Standard (RS) Two general dentists (A.K, A.N) independently evaluated the testing dataset and marked carious lesions using a custom labeling tool called CarioLabel, that automatically generated excel files [22]. Discrepancies between their assessments were reviewed and resolved by a specialist endodontist (N.N), thereby establishing a RS. Statistical analysis Quantitative analysis For all three models (ChatGPT-4o, Gemini 2.5 Flash and CNN), excel files were created manually by entering the findings from the models’ summary tables into CarioLabel, as these systems do not automatically produce excel outputs. The performance metrics were then computed against a RS using a custom Python code. Metrics such as accuracy, sensitivity, specificity, Positive Predictive Value (PPV), Negative Predictive Value (NPV), precision and F-1 score for caries detection on intraoral images were reported. Furthermore, to evaluate the difference in diagnostic performance between the models, a McNemar test was applied, with a p-value less than 0.05 considered statistically significant. Descriptive qualitative analysis For descriptive qualitative analysis, a structured Google Form (Google LLC, USA) was developed that included the original set of 22 intraoral images alongside their corresponding versions annotated with bounding boxes generated by all three models. The form incorporated a custom evaluation tool designed by the authors to assess the image output quality of LLMs, as no existing validated instrument or assessment criterion is available in literature. The complete evaluation tool is provided in Supplementary file 3. All images were anonymized and renamed using neutral labels (e.g., Image_01_A, Image_01_B, Image_01_C) to conceal the identity of the model. The images were presented in a randomized and blinded format to ensure that the two evaluators, both specialist endodontists (N.A. and S.S.) with five years of clinical experience, remained unaware of the generating model throughout the assessment. Each image was evaluated based on four predefined parameters: Realism of the output – The image looked natural, not distorted or confusing, and did not show supernumerary or additional features.Accuracy of decay identification – Carious lesion(s) were correctly marked.Bounding box precision – Boundaries completely enclosed the tooth with decay.Absence of unnecessary annotations – No sound enamel, restorations, or gingiva were wrongly marked. Realism of the output – The image looked natural, not distorted or confusing, and did not show supernumerary or additional features. Realism of the output – The image looked natural, not distorted or confusing, and did not show supernumerary or additional features. Accuracy of decay identification – Carious lesion(s) were correctly marked. Accuracy of decay identification – Carious lesion(s) were correctly marked. Bounding box precision – Boundaries completely enclosed the tooth with decay. Bounding box precision – Boundaries completely enclosed the tooth with decay. Absence of unnecessary annotations – No sound enamel, restorations, or gingiva were wrongly marked. Absence of unnecessary annotations – No sound enamel, restorations, or gingiva were wrongly marked. Evaluations were recorded using a 3-point Likert scale (1 = Disagree, 2 = Partially agree, 3 = Completely agree). For the ease of readers, the responses are referred to in the manuscript as (1 = poor, 2 = fair, and 3 = good). Responses collected through Google Forms were exported as a CSV file and a qualitative analysis was performed. For all three models, the qualitative evaluation was conducted using excel by combining the scores assigned by both evaluators for each assessment parameter, and the most frequently occurring score (mode) was reported as shown in Supplementary file 4. Results Quantitative analysis The quantitative analysis showed that ChatGPT-4o achieved an accuracy of 92.8%, sensitivity of 66.2%, specificity of 96.4%, PPV of 71.4%, NPV of 95.8%, 71.4% precision and F-1 score of 68.7% when compared with RS. In comparison, Gemini 2.5 Flash demonstrated an accuracy of 93.7%, sensitivity of 76.4%, specificity of 96.0%, PPV of 72.2%, NPV of 96.8%. precision of 72.2 and F-1 score of 74.3%. While CNN showed an accuracy of 97.2%, sensitivity of 86.7%, specificity of 98.6%, PPV of 89.3%, NPV of 98.2%, precision of 89.3% and F-1 score of 88.0%, as shown in Table 1. Table 1Performance metrics of ChatGPT-4o, Gemini 2.5 Flash and CNN compared to RSLLMsAccuracy in % (CI)Sensitivity in % (CI)Specificity in % (CI)PPVin % (CI)NPVin % (CI)Precisionin % (CI)F-1 score in % (CI)ChatGPT-4o92.8(90.7–94.9)66.2(54.9–77.4)96.4(94.8–98.0)71.4(60.3–82.6)95.8(93.8–97.4)71.4(60.2–82.5)68.7(64.9–72.5)Gemini 2.5 Flash93.7(91.7–95.6)76.4(66.3–86.5)96.0(94.3–97.7)72.2(61.8–82.5)96.8(95.3–98.3)72.2(61.8–82.5)74.3(70.7–77.8)CNN97.2(95.8–98.5)86.7(78.7–94.8)98.6(97.6–99.6)89.3(81.9–96.8)98.2(97.1–99.4)89.3(81.9–96.8)88.0(85.4–90.7)LLM Large Language Model, CNN Convolutional Neural Network, PPV Positive Predictive Value, NPV Negative Predictive Value, RS Reference Standard Table 1Performance metrics of ChatGPT-4o, Gemini 2.5 Flash and CNN compared to RSLLMsAccuracy in % (CI)Sensitivity in % (CI)Specificity in % (CI)PPVin % (CI)NPVin % (CI)Precisionin % (CI)F-1 score in % (CI)ChatGPT-4o92.8(90.7–94.9)66.2(54.9–77.4)96.4(94.8–98.0)71.4(60.3–82.6)95.8(93.8–97.4)71.4(60.2–82.5)68.7(64.9–72.5)Gemini 2.5 Flash93.7(91.7–95.6)76.4(66.3–86.5)96.0(94.3–97.7)72.2(61.8–82.5)96.8(95.3–98.3)72.2(61.8–82.5)74.3(70.7–77.8)CNN97.2(95.8–98.5)86.7(78.7–94.8)98.6(97.6–99.6)89.3(81.9–96.8)98.2(97.1–99.4)89.3(81.9–96.8)88.0(85.4–90.7)LLM Large Language Model, CNN Convolutional Neural Network, PPV Positive Predictive Value, NPV Negative Predictive Value, RS Reference Standard Performance metrics of ChatGPT-4o, Gemini 2.5 Flash and CNN compared to RS 92.8 (90.7–94.9) 66.2 (54.9–77.4) 96.4 (94.8–98.0) 71.4 (60.3–82.6) 95.8 (93.8–97.4) 71.4 (60.2–82.5) 68.7 (64.9–72.5) 93.7 (91.7–95.6) 76.4 (66.3–86.5) 96.0 (94.3–97.7) 72.2 (61.8–82.5) 96.8 (95.3–98.3) 72.2 (61.8–82.5) 74.3 (70.7–77.8) 97.2 (95.8–98.5) 86.7 (78.7–94.8) 98.6 (97.6–99.6) 89.3 (81.9–96.8) 98.2 (97.1–99.4) 89.3 (81.9–96.8) 88.0 (85.4–90.7) LLM Large Language Model, CNN Convolutional Neural Network, PPV Positive Predictive Value, NPV Negative Predictive Value, RS Reference Standard McNemar’s test revealed no statistically significant difference in diagnostic performance between ChatGPT-4o and Gemini 2.5 Flash (p-value = 0.541). However, CNN outperformed both models in terms of diagnostic performance showing statistically significant differences (p-values, ChatGPT-4o < 0.001; Gemini 2.5 Flash = 0.001) (Table 2). Table 2Statistical comparison of diagnostic performance between models using McNemar’s testModel 1Model 2p-value*ChatGPT-4oGemini 2.5 Flash0.541ChatGPT-4oCNN 0.0001* Gemini 2.5 FlashCNN 0.0016* CNN Convolutional Neural Network*p-value < 0.05 = significant Table 2Statistical comparison of diagnostic performance between models using McNemar’s testModel 1Model 2p-value*ChatGPT-4oGemini 2.5 Flash0.541ChatGPT-4oCNN 0.0001* Gemini 2.5 FlashCNN 0.0016* CNN Convolutional Neural Network*p-value < 0.05 = significant Statistical comparison of diagnostic performance between models using McNemar’s test CNN Convolutional Neural Network *p-value < 0.05 = significant Descriptive qualitative findings The results of the qualitative analysis are descriptively reported in Table 3. The radar chart in Fig. 2 displays four evaluation parameters at the four ends of the axes. The numerical scale represents the level of agreement, where a score of 1 indicates poor agreement, 2 indicates fair agreement, and 3 shows good agreement. Each color in the radar chart corresponds to one of the models. Table 3Descriptive qualitative evaluation of the models across four parameters on a three-point Likert scaleModelScoreRealismAccuracy of decay detectionBounding box precisionAbsence of unnecessary annotations n (%) n (%) n (%) n (%) ChatGPT-4o11 (2.2)22 (50.0)27 (61.3)29 (65.9)24 (9.0)14 (31.8)8 (18.1)1 (2.3)339 (88.6)8 (18.1)9 (20.4)14 (31.8)Gemini 2.5 Flash16 (13.6)8 (18.1)16 (36.3)28 (63.6)20 (0.0)20 (45.4)16 (36.3)3 (6.81)338 (86.3)16 (36.3)12 (27.2)13 (29.5)CNN10 (0.0)0 (0.0)0 (0.0)0 (0.0)24 (9.0)9 (20.4)3 (6.8)2 (4.5)340 (90.9)35 (79.5)41 (93.1)42 (95.4)CNN Convolutional Neural Network, 1: Poor, 2: Fair, 3: Good, n: Frequency of response Table 3Descriptive qualitative evaluation of the models across four parameters on a three-point Likert scaleModelScoreRealismAccuracy of decay detectionBounding box precisionAbsence of unnecessary annotations n (%) n (%) n (%) n (%) ChatGPT-4o11 (2.2)22 (50.0)27 (61.3)29 (65.9)24 (9.0)14 (31.8)8 (18.1)1 (2.3)339 (88.6)8 (18.1)9 (20.4)14 (31.8)Gemini 2.5 Flash16 (13.6)8 (18.1)16 (36.3)28 (63.6)20 (0.0)20 (45.4)16 (36.3)3 (6.81)338 (86.3)16 (36.3)12 (27.2)13 (29.5)CNN10 (0.0)0 (0.0)0 (0.0)0 (0.0)24 (9.0)9 (20.4)3 (6.8)2 (4.5)340 (90.9)35 (79.5)41 (93.1)42 (95.4)CNN Convolutional Neural Network, 1: Poor, 2: Fair, 3: Good, n: Frequency of response Descriptive qualitative evaluation of the models across four parameters on a three-point Likert scale CNN Convolutional Neural Network, 1: Poor, 2: Fair, 3: Good, n: Frequency of response Fig. 2Radar chart illustrating the comparison of ChatGPT-4o, Gemini 2.5 Flash, and the CNN model across all four parameters. CNN: Convolutional Neural Network Radar chart illustrating the comparison of ChatGPT-4o, Gemini 2.5 Flash, and the CNN model across all four parameters. CNN: Convolutional Neural Network All three models achieved good agreement for realism of the output, with percentage scores of ChatGPT-4o (88.6%), Gemini 2.5 Flash (86.3%), and CNN (90.9%). For accuracy of decay detection, only CNN achieved good agreement (79.5%). Again, the CNN achieved good agreement for both bounding box precision (93.1%) and absence of unnecessary annotations (95.4%). Remaining parameter scores are shown in Table 3. Discussion The integration of AI into dental diagnostics is increasingly being viewed as a promising approach for enhancing clinical decision-making and improving access to oral care [23]. To evaluate whether such tools can reliably support diagnostic workflows, this study compared the performance of two multimodal LLMs, ChatGPT-4o and Gemini 2.5 Flash, with a custom CNN for detecting and localizing dental caries on intraoral images. The aim was to determine whether these LLMs, without any domain-specific training, can approach the performance of a specialized CNN that requires large datasets, extensive computational resources, and task-focused optimization. While a recent study utilized fine-tuned vision-language models such as Florence-2 and PaLI-Gemma to identify carious lesions, the present work differs in two key aspects [24]. First, the models employed in this study, ChatGPT-4o and Gemini 2.5 Flash, are widely accessible to both clinicians and patients, unlike the ones used in prior work. Second, unlike the previous study, where carious lesions were only identified and localized without expert evaluation, the LLMs in the current study were also evaluated based on expert qualitative assessment. The results of the present study revealed that CNN outperformed LLMs in the quantitative analysis. It is important to note that multimodal LLMs are primarily conversation-based models and are not inherently designed to detect all lesions with accurate borders. In contrast, CNN-based architectures perform pixel-level object detection and are optimized for precise localization tasks. Despite this fundamental difference, we evaluated both model types using two approaches, a quantitative diagnostic assessment and a descriptive qualitative evaluation. This allowed a fair comparison of both models’ clinical interpretability and diagnostic utility across text-based detection and image-based annotation performance. The superior performance of CNNs in quantitative analysis could be attributed to its architecture, which is inherently optimized for visual feature extraction and pattern recognition [25]. This capability arises from their training on large, domain-specific datasets that allow them to learn fine-grained visual patterns relevant to dental pathology [26]. Consistent with this, previous studies have demonstrated that CNNs can achieve localization accuracies exceeding 90% in dental diagnostics [6, 27]. Although the CNN-based model achieved the highest overall performance, ChatGPT-4o and Gemini 2.5 Flash also demonstrated reasonably high accuracy in text-based outputs. Since this analysis focused on text-based tooth-level identification of caries rather than pixel-level localization, it favored models with strong visual reasoning and semantic interpretation, areas in which multimodal LLMs perform well [28]. These models detect decay by combining broad visual cues, tooth morphology, and previously learned clinical patterns, allowing them to correctly identify affected teeth even when precise spatial localization is imperfect [29]. Thus, in text-based, tooth-level evaluations, LLMs can achieve high accuracy [30, 31]. While both LLMs demonstrated comparable performance across all metrics, Gemini 2.5 Flash achieved higher sensitivity and a better F1-score than ChatGPT-4o. This difference may be attributed to architectural distinctions between the two. Gemini 2.5 Flash is designed as a natively multimodal model that jointly processes visual and textual features [32]. This enables it to detect subtle cues such as early enamel discoloration, or mild surface texture changes with more detail [32]. In contrast, ChatGPT-4o, although multimodal, is built on a text-first foundation, with visual capabilities integrated on top of the language model [29]. This architecture tends to interpret low-contrast or borderline visual features more conservatively, not identifying unless the lesion is visually obvious [33]. This can lead to fewer true-positive detections and, consequently, lower sensitivity. An interesting observation was that while the accuracies of all the models were high, their corresponding F-1 scores were comparatively low. This discrepancy is a classic example of the accuracy paradox in medical diagnostics, where high accuracy can be misleading if the model achieves it simply by correctly identifying the high volume of healthy (true negatives) teeth while missing a significant portion of diseased (true positive) ones [34]. For instance, while ChatGPT-4o and Gemini 2.5 Flash achieved accuracies above 92%, their lower F-1 scores reflect a struggle to balance precision and sensitivity. This suggests that while these models are adept at recognizing healthy tooth structures, they lack the diagnostic “aggressiveness” required to consistently detect actual carious lesions. This may also be attributed to the class imbalance within the testing dataset, which contained a higher number of healthy teeth compared to diseased ones. The descriptive qualitative evaluation in the study primarily focused on the realism of outputs, accuracy and localization of decay identification and the avoidance of unnecessary markings. The latter three parameters were used to evaluate image interpretation quality while realism was introduced after an unexpected observation during prompt testing. Although the prompt instructed the LLMs to return the original image with markings, both LLMs occasionally produced synthetic visual outputs. These output images contained fabricated teeth and artificially created carious lesions, as illustrated in the Supplementary file 5. This raises significant ethical and clinical concerns. Such synthetic images, if presented to clinicians, could create diagnostic confusion or misinterpretation, potentially impacting clinical decision-making. Similarly, if shown to patients, might provoke unnecessary anxiety or undermine trust in professional diagnosis [35]. Therefore, incorporating realism for descriptive qualitative evaluation was essential to ensure that AI-generated visual information remains clinically credible, safe and contextually appropriate. Nevertheless, these occurrences were infrequent, and were not reflected in the final result, as the evaluation considered the most frequently repeated responses. Consequently, the realism scores among all models were comparable, with no significant difference observed. However, notable differences were observed in the image interpretation parameters. Both ChatGPT-4o and Gemini 2.5 Flash frequently failed to identify carious sites accurately, often generating bounding boxes on sound teeth or normal anatomical structures. Despite these errors, Gemini 2.5 Flash demonstrated slightly better performance overall. In contrast, the custom CNN achieved complete concordance across all three parameters. These differences likely stem from LLMs being trained on broad, general-purpose image and/or text datasets rather than on dental-specific, pixel-level annotations [36]. Consequently, their visual grounding is less precise, particularly for tasks requiring fine discrimination of subtle intraoral features as supported by findings from a recent study [18, 36]. Both the quantitative and qualitative findings further highlighted the strength of the models. Since the LLMs demonstrated reasonable accuracy in textual diagnostic output (quantitative analysis), they may serve as valuable tools for patient education, clinical summarization, and automated report generation [36]. However, their limitations in precise localization (qualitative analysis) and annotation indicates that they are not yet suitable as standalone systems for image-based diagnosis [24]. In contrast, CNNs remain more reliable for caries detection and localization, making them more appropriate for integration into clinical workflows [6, 37]. Prospective and multi-center studies carried out more recently have gone beyond using AI models to detect carious lesions and examined their usefulness in other areas related to diagnostics and clinical decision-making. Evidence suggests that advanced LLMs can achieve near-expert-level performance in ranking oral lesions during differential diagnoses, while AI systems have demonstrated comparable outcomes to human experts in digital smile design [38, 39]. Similarly, a study reported that AI models could be on par with human expert judgment when interpreting periapical pathology on radiographs [40]. It appears that, while domain-specific AI models like CNNs prevail in the task of image localization, general-purpose models are catching up fast. A key strength of this study lies in its comprehensive methodological design. To the best of authors’ knowledge, this is the first investigation to compare a domain-specific CNN with multimodal LLMs for dental caries detection, using both textual and image-based outputs. The inclusion of a rigorous evaluation strategy, involving blinded scoring by specialists, further enhances the reliability of the findings. This study also had certain limitations. First, the LLMs used in the study were publicly available, free-tier versions that lack the enhanced multimodal reasoning capabilities of premium or API-level models. Free-tier versions often provide weaker visual interpretation, whereas paid versions offer more structured and context-aware analysis. Nonetheless, their use is justified, as these versions are the ones mostly accessible to patients and general users. Although the unit of analysis in this study was the individual tooth, some degree of intra-image correlation may exist because multiple teeth were evaluated from the same image. This could introduce partial dependence between observations, which may affect statistical estimates. However, given that dental caries is inherently a tooth-level condition and each tooth was assessed independently under standardized conditions, this approach was considered appropriate for the study objective. Nonetheless, this remains a methodological limitation. Future studies should consider applying clustered statistical approaches, such as Generalized Estimating Equations (GEE) or mixed-effects models, to better account for potential intra-image dependencies. A further limitation is that the qualitative assessment tool was specifically developed for this study and has not undergone formal validation; therefore, the qualitative findings should be interpreted as exploratory, reflecting preliminary insights rather than definitive or generalizable conclusions. Future research should include larger and more diverse intraoral datasets and evaluate models using API-level versions. Additionally, studies may explore hybrid approaches that combine CNNs for lesion localization with LLMs for diagnostic explanations. Moreover, given the strong performance of CNN-based systems demonstrated in this study and previous literature, the authors suggest that such models should be made publicly accessible through secure, web-based platforms. This will promote access to both patients and clinicians, ensuring that effective technologies are not underutilized. Conclusion The CNN outperformed multimodal LLMs in detecting and localizing dental caries on intraoral images. While LLMs demonstrated strong text-based tooth-level assessment and interpretive capabilities, their localization accuracy remained limited. These findings highlight CNNs as the benchmark for image-based diagnostics, with LLMs serving as valuable adjuncts for explanation, reporting, and patient communication. Integrating CNN-driven localization with LLM-based reasoning could form a powerful hybrid approach, advancing automated dental diagnostics and bridging the gap between technical accuracy and interpretive insight. Supplementary Information Supplementary Material 1. Supplementary Material 1. Supplementary Material 2. Supplementary Material 2. Supplementary Material 3. Supplementary Material 3. Supplementary Material 4. Supplementary Material 4. Supplementary Material 5. Supplementary Material 5.