GlobalDentBench: A Multinational Benchmark for Evaluating LLM Clinical Reasoning in Dentistry with Expert Calibration

Junjie Zhao, Jingyi Liang, Zhenyang Cai, Jiaming Zhang, Zhenwei Wen, Shuzhi Deng, Wenjing Yi, Chunfeng Luo, Hexian Zhang, Junying Chen, Tianrui Liu, Zhuhui Bai, Zixu Zhang, Pradeep Singh, Xiang Liu, Jianquan Li, Nhan L Tran, Falk Schwendicke, Zuolin Jin, Lijian Jin, Liangyi Chen, Wei-fa Yang, Benyou Wang, Junwen Wang, Shan Jiang · 2026

Read GlobalDentBench: A Multinational Benchmark for Evaluating LLM Clinical Reasoning in Dentistry with Expert Calibration by Junjie Zhao, Jingyi Liang, Zhenyang Cai, Jiaming Zhang, Zhenwei Wen, Shuzhi Deng, Wenjing Yi, Chunfeng Luo, Hexian Zhang, Junying Chen, Tianrui Liu, Zhuhui Bai, Zixu Zhang, Pradeep Singh, Xiang Liu, Jianquan Li, Nhan L Tran, Falk Schwendicke, Zuolin Jin, Lijian Jin, Liangyi Chen, Wei-fa Yang, Benyou Wang, Junwen Wang, Shan Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24636.txt.

Introduction

Junjie Zhao and colleagues introduce GlobalDentBench, the first multinational dental benchmark, spanning 14 specialties across 88 countries and six continents. Its 8,978 expert-validated questions cover multiple-choice, short-answer, and case-based formats at three reasoning levels, from recall to individualized reasoning. Six senior dentists calibrated the construction, with agreement near 100 percent on simpler formats and 96.78 percent on case-based questions. Twelve frontier models degrade stepwise with complexity, and 31.01 percent of real-case recommendations were unsafe, with 4.51 percent risking irreversible harm.

Catalog record

CREATORJunjie Zhao, Jingyi Liang, Zhenyang Cai, Jiaming Zhang, Zhenwei Wen, Shuzhi Deng, Wenjing Yi, Chunfeng Luo, Hexian Zhang, Junying Chen, Tianrui Liu, Zhuhui Bai, Zixu Zhang, Pradeep Singh, Xiang Liu, Jianquan Li, Nhan L Tran, Falk Schwendicke, Zuolin Jin, Lijian Jin, Liangyi Chen, Wei-fa Yang, Benyou Wang, Junwen Wang, Shan Jiang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74940
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24636

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GlobalDentBench: A Multinational Benchmark for Evaluating LLM Clinical Reasoning in Dentistry with Expert Calibration Junjie Zhao, Jingyi Liang, Zhenyang Cai, Jiaming Zhang, Zhenwei Wen, Shuzhi Deng, Wenjing Yi, Chunfeng Luo, Hexian Zhang, Junying Chen, Tianrui Liu, Zhuhui Bai, Zixu Zhang, Pradeep Singh, Xiang Liu, Jianquan Li, Nhan L Tran, Falk Schwendicke, Zuolin Jin, Lijian Jin, Liangyi Chen, Wei-fa Yang, Benyou Wang, Junwen Wang, Shan Jiang Abstract While large language models (LLMs) hold transformative potential for medicine, their reasoning robustness and safety in real-world clinical scenarios remain critically underexplored, particularly in dentistry. Here we introduce GlobalDentBench, the first multinational dental benchmark, featuring a taxonomy that encompasses 14 dental specialties across 88 countries and regions spanning six continents. The benchmark comprises 8,978 expert-validated questions across three formats (multiple-choice, short-answer, and case-based questions) and assesses three progressive reasoning levels: knowledge recall (L1), routine reasoning (L2), and individualized reasoning (L3). To ensure data quality, the automated construction framework was calibrated by six senior dentists, achieving expert agreement rates of 99.98% for multiple-choice and short-answer questions and 96.78% for the more complex case-based questions. Evaluation of 12 frontier LLMs on GlobalDentBench revealed a sharp, stepwise performance degradation with increasing reasoning complexity. Specifically, accuracy plummeted from 81.34% on multiple-choice to 64.53% on short-answer and 22.34% on case-based questions, while declining markedly from 74.01% at L1 to 55.64% at L2 and 35.71% at L3. More critically, risk analysis of real-world dental cases demonstrated an alarming overall unsafe rate of 31.01% in LLM-generated clinical recommendations, with 4.51% posing risks of irreversible patient harm and risks particularly pronounced in specialties such as orthodontics. These findings expose fundamental limitations in the medical reasoning and safety of current LLMs. Consequently, GlobalDentBench provides a scalable foundation for trustworthy clinical AI evaluation, underscoring the urgent need for rigorous validation before the safe deployment of these models in healthcare. 1 Division of Applied Oral Sciences and Community Dental Care, Faculty of Dentistry, The University of Hong Kong. 2 School of Data Science, The Chinese University of Hong Kong, Shenzhen. 3 School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen. 4 Department of Periodontology, Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University, Shenzhen, China. 5 Beijing Institute of Collaborative Innovation. 6 Department of Orthodontics, Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University, Shenzhen, China. 7 Shenzhen Stomatology Hospital (Pingshan) of Southern Medical University, Shenzhen, China. 8 College of Future Technology, Peking University. 9 Freedom AI. 10 New Cornerstone Science Laboratory, National Biomedical Imaging Center, State Key Laboratory of Membrane Biology, Institute of Molecular Medicine, Peking-Tsinghua Center for Life Sciences, College of Future Technology, Peking University, Beijing 100871, China; 11 IDG/McGovern Institute for Brain Research, Peking University, Beijing 100871, China. 12 Division of Oral and Maxillofacial Surgery, Faculty of Dentistry, The University of Hong Kong. 13 Shenzhen Loop Area Institute. 14 Department of Cancer Biology, Mayo Clinic Arizona, 5777 E. Mayo Blvd., IERB-3-504A, Phoenix, Arizona, 85054, USA. 15 Department of Conservative Dentistry, Periodontology and Digital Dentistry, LMU University Hospital, LMU Munich, Munich, Germany. 16 Division of Periodontology & Implant Dentistry, Faculty of Dentistry, The University of Hong Kong, Hong Kong, SAR, China \dagger Equal Contributions * Corresponding authors. [email protected](L.C.), [email protected] (W.-f.Y.), [email protected] (B.W.), [email protected] (J.W.), [email protected] (S.J.) 1 Introduction The rise of large language models (LLMs) has initiated a paradigm shift for artificial intelligence within healthcare, offering sophisticated capabilities for the understanding of natural language, the synthesis of knowledge, and the support of clinical decisions [ 12 , 34 , 20 ] . Through extensive training upon vast corpora of medical literature and clinical data, frontier models have demonstrated a high level of proficiency in standardized examinations for medical licensing, frequently matching the performance of human experts within controlled settings [ 36 , 37 ] . Systematic evaluations across various branches of medicine indicate that these models excel in communication with patients, health education, and scenarios involving simulated diagnosis, thereby offering the potential to alleviate the cognitive burden on practitioners while supporting practice rooted in evidence [ 16 ] . Nevertheless, a persistent gap remains between the performance observed on benchmarks modeled after closed examinations and the rigorous requirements of clinical environments in the real world. These authentic settings necessitate the management of diagnostic ambiguity, the processing of incomplete information, and the execution of iterative reasoning across multiple steps under conditions of uncertainty [ 33 , 52 , 22 ] . This disparity is particularly acute within the field of dentistry, a discipline requiring the integration of spatial analysis, procedural expertise, and treatment planning over longitudinal periods, all tailored to the context of individual patients [ 47 , 6 ] . Although LLMs have been utilized for dental queries and educational tasks, existing assessments are largely restricted to examinations for local licensing or narrow subfields. These evaluations rely predominantly on multiple-choice questions(MCQs) that measure the recall of facts rather than the inferential reasoning based on specific cases that remains essential to routine clinical practice [ 13 , 5 ] . Studies focused on benchmarking consistently demonstrate that the performance of models declines sharply when tasks transition from the retrieval of canonical knowledge to open reasoning that incorporates clinical ambiguity [ 1 , 39 , 27 ] . Recent progress in multimodal models, such as DentalGPT [ 9 ] , ToothXpert [ 24 ] , OralGPT [ 21 ] and DentVLM [ 28 ] represents the current technological frontier for artificial intelligence specialized for dentistry. These systems have achieved notable milestones in integrating visual and linguistic data for the purpose of complex reasoning. However, such advances also illuminate the limitations of current evaluation frameworks, which remain fragmented and insufficiently aligned with the full spectrum of dental practice across diverse geographies. This underscores an urgent requirement for a more rigorous and globally representative benchmark capable of systematically measuring the robustness of reasoning under authentic clinical conditions [ 25 , 26 ] . Foundational benchmarks for medical models, while useful for the establishment of initial capabilities, possess inherent constraints that limit their applicability in a clinical setting [ 41 , 44 ] . In domains where the stakes are high, models exhibit documented weaknesses, including an overconfidence in incorrect outputs, sensitivity to variations in prompts, and the occurrence of hallucinations during reasoning across multiple steps [ 8 , 50 ] . Such limitations carry direct consequences for clinical deployment; evaluations of collaboration between physicians and AI show that models may introduce cognitive biases if close supervision is absent [ 32 , 35 ] . In dentistry, where errors in diagnosis can produce consequences that are irreversible, the lack of a framework for evaluation spanning multiple levels of cognition has restricted the reliable assessment of whether these models are ready for clinical use [ 14 , 15 ] . Here we introduce GlobalDentBench, the first multinational benchmark designed to evaluate the robustness of reasoning within dentistry. Spanning 88 countries or regions and 14 dental specialties, GlobalDentBench comprises 8,978 questions validated by experts. These questions are stratified across three clinically anchored reasoning levels: L1 (Knowledge Recall), L2 (Routine Reasoning), and L3 (Individualized Reasoning). Source materials are drawn exclusively from qualification examinations of a gold standard, textbooks subjected to peer review, and case reports of high impact, ensuring direct alignment

Read the complete text