Prompting language influences diagnostic reasoning and accuracy of large language models
Adrien Bazoge, Josselin Corvellec, Pierre-Antoine Gourraud · 2026
Read Prompting language influences diagnostic reasoning and accuracy of large language models by Adrien Bazoge, Josselin Corvellec, Pierre-Antoine Gourraud (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19173.txt.
Introduction
Adrien Bazoge and three colleagues test whether prompting language changes diagnostic thinking and accuracy in language models, comparing English and French across five models. Two doctors scored 180 cases across sixteen fields on an eighteen-point scale for diagnosis and thinking quality. Four of the five models scored higher in English, by 0.37 to 0.91 on average, with gaps in differential diagnosis, logical structure, and internal validity. The o3 model alone showed no language effect. Prompting language is thus a key driver of clinical results, with stakes for fair use globally.
Catalog record
| CREATOR | Adrien Bazoge, Josselin Corvellec, Pierre-Antoine Gourraud |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58737 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19173
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompting language influences diagnostic reasoning and accuracy of large language models Adrien Bazoge, Josselin Corvellec, Pierre-Antoine Gourraud Abstract Large language models (LLMs) are increasingly explored for clinical decision support, yet most evaluations are conducted in English, leaving their reliability in other languages uncertain. Here we evaluate the impact of prompting language on diagnostic reasoning and final diagnosis accuracy by comparing English and French performance across five LLMs (o3, DeepSeek-R1, GPT-4-Turbo, Llama-3.1-405B-Instruct, and BioMistral-7B). A total of 180 clinical vignettes covering 16 medical specialties were assessed by two physicians using an 18-point scale evaluating both diagnosis accuracy and reasoning quality. Four of the five models performed better in English (mean difference 0.37–0.91, adjusted p < < 0.05), with the gap spanning multiple aspects of reasoning, including differential diagnosis, logical structure, and internal validity. o3 was the only model showing no overall language effect. These findings demonstrate that prompting language remains a critical determinant of LLM clinical performance, with implications for equitable linguistico-cultural deployment worldwide. \keepXColumns 1 Introduction Among AI applications, the accessibility of Large Language Models (LLMs) is rapidly transforming many sectors including medicine, with promising applications in assisting clinicians in diagnostic reasoning and decision-making, reducing the burden of administrative tasks to free up medical time, and enabling people to easily access basic health advice [ 34 , 6 , 18 ] . By encoding vast clinical knowledge and analyzing complex information from electronic health records at scale and in near real time, LLMs have the potential to support clinicians across specialties and healthcare systems, while also improving access to health information and high-quality care, especially in low-resource settings [ 5 , 3 , 29 , 22 ] . Despite this potential, important challenges remain before LLMs can be responsibly integrated into clinical practice, ensuring that innovation aligns with Hippocratic values—prioritizing patient safety, autonomy, privacy, and equal opportunity for quality care [ 7 ] . Most models are trained predominantly on English-language data from high-income regions like the United States and Europe, and primarily benchmarked in English-speaking contexts, raising concerns about their reliability in multilingual and culturally diverse healthcare settings [ 9 , 20 ] . This is particularly acute in medicine, where language is inseparable from culture, epidemiology, clinical practice, and regulation. Substantial amounts of clinically relevant knowledge remain inaccessible, underrepresented, or unavailable in training corpora due to privacy constraints. Even when such knowledge is encoded, its effective use depends on its accurate interpretation and application within the user’s own linguistic, cultural, and healthcare context. Without robust multilingual and multicultural alignment at both the training and deployment stages, LLMs risk missing the opportunity to reduce inequities in access to trustworthy health information, reliable clinical decision support, and contextually appropriate care, especially for populations whose languages and healthcare realities are poorly represented in the data that shape these models. Furthermore, evaluations of LLMs mostly rely on simplified standardized and multiple-choice benchmarks [ 29 , 30 ] , which do not capture the complexity of real clinical routine practice. As a result, their evaluation in real-world clinical settings remains limited [ 18 , 5 ] , leaving uncertainties about their reliability and safety across diverse healthcare environments. Prior work has demonstrated the ability of LLMs to generate accurate diagnoses and differential diagnoses from simulated clinical vignettes and sequential clinical encounters, but most evaluations have been conducted exclusively in English [ 16 , 26 , 33 , 21 , 25 ] . The few studies that have explored multilingual evaluation in medical context were either limited in the number of cases [ 18 ] , restricted to multiple-choice format from medical examinations [ 4 , 24 , 32 , 36 ] , or did not explicitly analyze performance differences across languages [ 5 ] . Emerging evidence from these studies indicates that model performances can vary significantly across languages [ 18 , 4 , 24 , 32 , 36 , 15 ] , raising the question of whether performance on clinical decision tasks, including diagnostic reasoning, remains consistent in non-English contexts. To address this gap, we designed a bilingual English-French comparative evaluation framework of diagnostic reasoning and final diagnosis accuracy, as these represent the entry door of clinical decision-making and are crucial for delivering effective patient care [ 13 , 2 ] . Five LLMs spanning different architectures and capability levels were evaluated: o3 [ 14 ] , DeepSeek-R1 [ 12 ] , GPT-4-Turbo [ 23 ] , Llama-3.1-405B-Instruct (Llama-405B) [ 11 ] , and BioMistral-7B [ 17 ] . A total of 180 clinical vignettes covering 16 medical specialties and multiple diagnostic and reasoning types were independently evaluated by two physicians using an 18-point scale assessing both the accuracy of the final diagnosis and the quality of the underlying clinical reasoning. 2 Methods 2.1 Design of the vignettes This study investigates diagnostic reasoning and clinical diagnosis across medical and medical-surgical specialties. Disciplines with a primarily technical focus (radiology, clinical biology, anatomical pathology and nuclear medicine) and purely surgical fields were excluded. Sixteen specialties were retained, covering the breadth of clinical medicine: emergency and critical care, endocrinology and metabolism, gynecology, oncology-hematology, hepatology-gastroenterology, infectious and tropical diseases, cardiovascular medicine, general practice, internal medicine, neurology, head and neck medicine, pediatrics, pulmonology, psychiatry, rheumatology, and urology-nephrology. For each specialty, approximately 10 vignettes were developed (range: 6–17), except for general practice, which was the specialty of the two physicians, for which 32 vignettes were created, yielding a total of 180 vignettes. Each vignette was designed with a predefined expected diagnosis and contained all information necessary to establish it, including first-line ancillary test results when clinically appropriate. We adopted this vignette-based design to approximate the realities of early clinical encounters, such as those in outpatient or emergency care. In these settings, physicians usually work with a narrow set of focused questions, resulting in histories that may be sparse, incomplete, or occasionally contain irrelevant details. Vignettes were derived from three sources: (1) synthetic vignettes created de novo by physicians in accordance with current medical guidelines, (2) vignettes adapted from reference materials (textbooks, lectures, and residency training resources), and (3) non-identifying details based on real-world clinical encounters. Vignettes adapted from reference texts were reformulated into vignette format to meet predefined criteria and to minimize potential overlap with models’ training data available online. For synthetic and real vignettes, final diagnoses were proposed by physicians; for vignettes based on reference texts, diagnoses were extracted from the source. Diagnoses were considered the expected diagnosis, regardless of certainty, and reflected the most relevant, probable, or clinically actionable condition. For example, in a critically ill patient, the expected diagnosis was septic shock rather than identification of the specific pathogen. Depending on context, diagnoses could be etiological ( n = 113), syndromic ( n = 54), or paraclinical ( n = 13). For all vignettes, physicians provided a reference diagnostic reasoning pathway leading to the final diagnosis. Reasoning processes were categorized into five types, each being assigned a single predominant type for analytical purposes, although clinical reasoning in practice often involves multiple simultaneous approaches: • Case recognition ( n = 57): non-analytical reasoning based on pattern recognition or recall of previously encountered cases; effective for simple and typical cases, and requiring a solid clinical background. •