Multimodal foundation models exploit text to make medical image predictions Authors: Thomas A. Buckley, James A. Diao, Cam N. Srivastava, Peter G. Brodeur, Pranav Rajpurkar, Adam Rodman, Arjun K. Manrai Published: 2026 ABSTRACT Multimodal foundation models have shown compelling but conflicting performance in medical image interpretation. However, the ways in which these models integrate and prioritize different data modalities, including images and text, remain poorly understood. Here we evaluate 8 proprietary and open-source multimodal foundation models using 1090 multimodal medical cases. We show that image predictions are largely driven by text, with accuracy increasing monotonically with the amount of informative text. Exploitation of text is a double-edged sword; even mild suggestions of an incorrect diagnosis in text diminish image-based classification, dramatically reducing performance in cases the model could previously answer using images alone—o3 accuracy fell from 84% to 28% when a misleading clinical vignette was introduced. In physician evaluations of long-form cases, adding images reduces or does not improve performance when text is highly informative (e.g., GPT-4V showed decreased accuracy when images were added to highly informative text across 69 clinicopathological conferences). Our results suggest that multimodal AI models may be useful in medical diagnostic reasoning but that their accuracy is largely driven, for better and worse, by text. Introduction With the increasing complexity and density of clinical information, artificial intelligence (AI) applications in medicine have shown promise for assisting with clinical reasoning tasks. Most medical AI models are intended for a specific clinical task, such as detecting diabetic retinopathy from retinal fundus photographs1,2. While large language models (LLMs) like the Generative Pre-trained Transformer (GPT) series of models3,4 have demonstrated compelling performance in general medical reasoning, evaluations have largely focused on tasks using text alone, such as answering medical licensing exam questions5, writing empathetic responses to patient queries6, and solving challenging diagnostic cases7–9. Unlike many AI models that accept one data modality as inputs (unimodal), human experts utilize a variety of sensory inputs and data modalities to make clinical decisions (multimodal)10,11. There is optimism that multimodal foundation models—models trained using vast amounts of text and images—may exhibit superior general diagnostic performance compared to systems that use images or text alone. Existing evaluations of multimodal foundation models in medicine are compelling but conflicted, with some studies showing that even older multimodal models like GPT-4 outperformed human experts12,13 and others indicating poor performance on image-based medical tasks14,15. Moreover, the prioritization of images compared to text in multimodal diagnosis, and therefore model generalizability in clinical settings, remains poorly understood. Here, using a collection of 1090 multimodal cases, we evaluated the unimodal and multimodal medical image interpretation abilities of several general-purpose and clinical vision-language models, including OpenAI’s o3 reasoning model, GPT-516, GPT-4 with vision model (GPT-4V and GPT-4V Turbo)17,18, Meta’s open-source Llama-3.2-90B model19, Google’s Gemini Pro 1.020, MedGemma, and LLaVA-Med-v1.521. Using this diverse set of cases, we study the accuracy, failure modes, and reasoning abilities of multimodal foundation models. These cases include text prompts that vary considerably in their length and information content, enabling fine-grained analyses of the relative contributions to accuracy from images versus the text that accompanies them. Our study reveals an underlying pattern where vision-language models exploit informative text to make medical image predictions. Results Accuracy of multimodal foundation models on challenging medical cases compared to humans We measured the performance of foundation models when provided the question text and image (see Supplementary Information for prompt). The o3 reasoning model achieved an overall accuracy of 81% (79% to 84%) across 945 image challenge cases, compared to an overall accuracy of 49% (95% CI, 49% to 50%) for human respondents (Table 1). The GPT-5 model performs similarly to o3 (accuracy 84%, 95% CI, 82% to 87%). On a subset of 100 questions answered by 3 internal medicine physicians, the overall accuracy was 52% (95% CI, 40% to 64%). MedGemma, a model trained on clinical text and images, had an accuracy of 52% (95% CI, 49% to 55%). LLaVA-Med-1.5 had the lowest accuracy of 31% (95% CI, 28% to 34%). Because o3 was pre-trained on data through June 1, 2024, which may include earlier published cases, we compared its performance on 51 cases published after this cutoff with 335 cases published between 2018 and June 1, 2024 (cases with similar text length and respondent performance; Supplementary Figs. 1, 2). Accuracy was 88% (95% CI, 76% to 96%) for post-cutoff cases and 94% (95% CI, 91% to 96%) for pre-cutoff cases. Respondents had accuracies of 52% (95% CI, 49% to 55%) and 51% (95% CI, 49% to 52%) for the two groups, respectively. These results suggest minimal risk of accuracy inflation from data leakage.Table 1Overall resultsAll QuestionsModel NameOverall (n = 945)Radiology (n = 220)Dermatology (n = 310)Other (n = 415)GPT-5 (2025)84% (82% to 87%)74% (68% to 80%)87% (82% to 90%)88% (85% to 91%)o3 (2025)81% (79% to 84%)70% (64% to 76%)82% (77% to 86%)87% (83% to 90%)GPT-4V Turbo (2024)63% (60% to 66%)52% (45% to 59%)63% (57% to 68%)68% (64% to 73%)Llama-3.2-90B (2024)58% (55% to 61%)50% (43% to 57%)56% (50% to 61%)64% (59% to 68%)MedGemma (2025)52% (49% to 55%)42% (35% to 49%)52% (47% to 58%)58% (53% to 62%)Gemini Pro 1.0 (2024)41% (38% to 44%)37% (30% to 44%)36% (31% to 42%)47% (42% to 52%)LLaVA-Med-1.5 (2024)31% (28% to 34%)28% (22% to 35%)32% (27% to 37%)33% (28% to 38%)NEJM Respondents49% (49% to 50%)49% (47% to 51%)50% (48% to 51%)50% (48% to 51%)3 Internal Medicine Physicians52% (40% to 64%)–––Accuracy of LLMs and human respondents on 945 cases published between 2005 and 2023. The accuracy for NEJM Respondents is the mean of the proportion correct for each set of cases. The accuracy for vision-language models is the proportion of correct responses for each set of cases. 95% confidence intervals (95% CIs) for NEJM Respondents are computed from a t-distribution. 95% CIs for vision-language model accuracy are computed using Clopper-Pearson intervals. 95% confidence intervals for physician performance were computed using a two-way cluster bootstrap that resamples both physicians and cases with replacement to account for clustering at both levels. Table 1Overall resultsAll QuestionsModel NameOverall (n = 945)Radiology (n = 220)Dermatology (n = 310)Other (n = 415)GPT-5 (2025)84% (82% to 87%)74% (68% to 80%)87% (82% to 90%)88% (85% to 91%)o3 (2025)81% (79% to 84%)70% (64% to 76%)82% (77% to 86%)87% (83% to 90%)GPT-4V Turbo (2024)63% (60% to 66%)52% (45% to 59%)63% (57% to 68%)68% (64% to 73%)Llama-3.2-90B (2024)58% (55% to 61%)50% (43% to 57%)56% (50% to 61%)64% (59% to 68%)MedGemma (2025)52% (49% to 55%)42% (35% to 49%)52% (47% to 58%)58% (53% to 62%)Gemini Pro 1.0 (2024)41% (38% to 44%)37% (30% to 44%)36% (31% to 42%)47% (42% to 52%)LLaVA-Med-1.5 (2024)31% (28% to 34%)28% (22% to 35%)32% (27% to 37%)33% (28% to 38%)NEJM Respondents49% (49% to 50%)49% (47% to 51%)50% (48% to 51%)50% (48% to 51%)3 Internal Medicine Physicians52% (40% to 64%)–––Accuracy of LLMs and human respondents on 945 cases published between 2005 and 2023. The accuracy for NEJM Respondents is the mean of the proportion correct for each set of cases. The accuracy for vision-language models is the proportion of correct responses for each set of cases. 95% confidence intervals (95% CIs) for NEJM Respondents are computed from a t-distribution. 95% CIs for vision-language model accuracy are computed using Clopper-Pearson intervals. 95% confidence intervals for physician performance were computed using a two-way cluster bootstrap that resamples both physicians and cases with replacement to account for clustering at both levels. Overall results Accuracy of LLMs and human respondents on 945 cases published between 2005 and 2023. The accuracy for NEJM Respondents is the mean of the proportion correct for each set of cases. The accuracy for vision-language models is the proportion of correct responses for each set of cases. 95% confidence intervals (95% CIs) for NEJM Respondents are computed from a t-distribution. 95% CIs for vision-language model accuracy are computed using Clopper-Pearson intervals. 95% confidence intervals for physician performance were computed using a two-way cluster bootstrap that resamples both physicians and cases with replacement to account for clustering at both levels. The top performing models, GPT-5, o3, and GPT-4V Turbo, exhibited lower performance on radiology questions, compared to dermatology images (Table 1). We also analyzed the accuracy of AI models stratified by skin type (Supplementary Table 2), disease rarity (Supplementary Fig. 7), prompting strategy (Supplementary Table 7), and image resolution (Supplementary Figs. 9, 10), and found no significant differences. We found that the performance of o3 is correlated with the performance of respondents to the online NEJM poll (r = 0.26, p < 0.001, Supplementary Fig. 8). To further compare physician and model performance, we examined overlap in errors using confusion matrices (Supplementary Fig. 6). Multimodal reasoning over text, images, and both among several vision-language models For each case, we compared the performance of o3 (Multimodal, using both image and text inputs) to o3 (Text Only) and o3 (Image Only). We stratified performance by the decile of the question word count as a proxy for the information content (Fig. 1A). When the text is uninformative (e.g., What is the diagnosis?), o3 (Multimodal) matches the performance of o3 (Image Only). At the other extreme, when text is highly informative, o3 (Text Only) performs equally well as o3 (Multimodal), and substantially outperforms o3 (Image Only). Between these extremes, o3 (Multimodal) outperforms both its text-only and image-only counterparts.Fig. 1Unimodal and multimodal clinical reasoning by vision-language models.A Comparison of OpenAI o3 and Human Respondents Across Modalities Accuracy of OpenAI o3 when using images only, text only, or both on 945 NEJM Image Challenge cases compared to human respondents, stratified by the length of the case text. Bins are created by decile of question word count. Actual word count ranges are included in parentheses on the x-axis. B Performance of Vision-Language Models and Physicians with Increasing Amounts of Text Accuracy of vision-language models compared to 3 internal medicine physicians on a subset of 96 Image Challenge cases. 4 cases are missing from the original subset as they were outside of the LLM evaluation set. Questions were grouped into short and long text questions using the median word count. C Relative Improvement From Adding Images or Text, Stratified by Difficulty Relative change in accuracy between (1) image only to multimodal or (2) text only to multimodal on a subset of 711/945 Image Challenge cases that include text information. Performance is stratified by question difficulty, categorized using tertiles of human respondent performance. Confidence intervals were computed using bootstrapping with 5000 replicates. D Variability of OpenAI o3 by Question Word Count Results of running o3 20 times each on a random subset of 300 questions from the 945 Image Challenge cases. Variability was measured as 1 - max proportion, where max proportion is the highest proportion among the 5 answer choices selected over the 20 trials. We stratified variability by word count tertile. Unimodal and multimodal clinical reasoning by vision-language models. A Comparison of OpenAI o3 and Human Respondents Across Modalities Accuracy of OpenAI o3 when using images only, text only, or both on 945 NEJM Image Challenge cases compared to human respondents, stratified by the length of the case text. Bins are created by decile of question word count. Actual word count ranges are included in parentheses on the x-axis. B Performance of Vision-Language Models and Physicians with Increasing Amounts of Text Accuracy of vision-language models compared to 3 internal medicine physicians on a subset of 96 Image Challenge cases. 4 cases are missing from the original subset as they were outside of the LLM evaluation set. Questions were grouped into short and long text questions using the median word count. C Relative Improvement From Adding Images or Text, Stratified by Difficulty Relative change in accuracy between (1) image only to multimodal or (2) text only to multimodal on a subset of 711/945 Image Challenge cases that include text information. Performance is stratified by question difficulty, categorized using tertiles of human respondent performance. Confidence intervals were computed using bootstrapping with 5000 replicates. D Variability of OpenAI o3 by Question Word Count Results of running o3 20 times each on a random subset of 300 questions from the 945 Image Challenge cases. Variability was measured as 1 - max proportion, where max proportion is the highest proportion among the 5 answer choices selected over the 20 trials. We stratified variability by word count tertile. On a random subset of 96 questions that were also answered by 3 internal medicine physicians, we found that all models exhibited increased performance on questions with longer text (Fig. 1B). GPT-5 and o3 both score 70% on low-text questions that require direct image interpretation, but score 88% and 90% respectively on high-text questions (o3 absolute improvement 20%, 95% CI, 5% to 36%; GPT-5 absolute improvement 18%, 95% CI, 2% to 34%). Physicians did not improve with increased amounts of text, and had an accuracy of 49% on low-text questions and 54% on high-text questions (difference 4.2%, p = 0.69). MedGemma, GPT-4V Turbo, and Llama 3.2 90B perform significantly worse than physicians on questions with less text, but outperform them on questions with more text. Gemini Pro 1.0 and LLaVA-Med are inferior to physicians across text lengths. For GPT-4V Turbo and o3, we analyzed the relative improvements from adding either text or images to the case vignette, stratified by difficulty (Fig. 1C). We used a subset of 711/945 cases that had accompanying text beyond an uninformative question (e.g., “What is the diagnosis?”). Difficulty was defined using tertiles of NEJM respondent performance. For both o3 and GPT-4V Turbo, the addition of text results in a larger improvement in performance on difficult questions compared to easy ones: for GPT-4V Turbo, the relative performance increase for easy questions is 60% (95% CI, 42% to 83%), whereas the increase for difficult questions is 128% (95% CI, 79% to 200%). For o3, the relative performance increase for easy questions is 21% (95% CI, 13% to 31%), whereas the increase for difficult questions is 43% (95% CI, 28% to 61%). The addition of images compared to text alone showed a smaller improvement across difficulty levels. Finally, we ran the o3 model 20 times each on a random subset of 300 Image Challenge questions to assess the effect of text content on response variability. Figure 1d shows the variability across three tertiles of question length. Variability significantly decreases with an increased amount of text content (Spearman ρ = −0.29, p-value < 0.001). We observed that models performed significantly worse on cases published before January 2018 compared with those published later. In contrast, human respondent performance was not strongly correlated with publication year (Supplementary Fig. 2). Notably, earlier cases contained substantially less accompanying text than more recent ones (Supplementary Fig. 1), which likely contributed to reduced model performance. Vision-language models anchor to misleading text For the 348 cases that GPT-4V Turbo correctly answered using images alone, we assessed how adding misleading text would affect model performance. Two types of misleading text were tested: (1) a direct suggestion from a fictional colleague and (2) a fictional patient vignette generated by GPT-4 that implied an incorrect disease. As shown in Fig. 2, the colleague’s suggestion reduced accuracy from 84% (95% CI, 80% to 88%) to 78% (95% CI, 75% to 80%) for o3, and from 100% (95% CI, 99% to 100%) to 27% (95% CI, 24% to 29%) for GPT-4V Turbo. Notably, GPT-4V Turbo followed the incorrect option suggested by the fictional colleague in 68% (95% CI, 66% to 71%) of cases, whereas o3 did so in only 13% (95% CI, 11%–14%). The fictional patient vignette had an even stronger effect. Accuracy fell to 6.8% (95% CI, 5.5%–8.3%) for GPT-4V Turbo and 28% (95% CI, 26%–30%) for o3, with both models most often selecting the incorrect disease suggested in the vignette.Fig. 2Susceptibility of vision-language models to misleading text.Using 348 Image Challenges correctly answered by GPT-4V Turbo with the image alone, we evaluated the robustness of GPT-4V Turbo and o3 to either suggestions from a fictional colleague or fictional vignettes suggestive of another disease. We added misleading text for all 4 incorrect options for each quiz, resulting in 1392 responses. We measured the proportion of responses in which the model was (1) completely incorrect, (2) incorrect, but chose the disease suggested by the text, or (3) correct. For colleague suggestions, we added the text: “Your esteemed colleague has taken a history of present illness, suggesting that the patient’s symptoms closely align with {distract_diagnosis}.” To generate fictional vignettes, we used each incorrect quiz option and prompted GPT-4: “Create a fictional patient vignette that strongly suggests the following: {diagnosis}” (see Supplement for full prompts). Susceptibility of vision-language models to misleading text. Using 348 Image Challenges correctly answered by GPT-4V Turbo with the image alone, we evaluated the robustness of GPT-4V Turbo and o3 to either suggestions from a fictional colleague or fictional vignettes suggestive of another disease. We added misleading text for all 4 incorrect options for each quiz, resulting in 1392 responses. We measured the proportion of responses in which the model was (1) completely incorrect, (2) incorrect, but chose the disease suggested by the text, or (3) correct. For colleague suggestions, we added the text: “Your esteemed colleague has taken a history of present illness, suggesting that the patient’s symptoms closely align with {distract_diagnosis}.” To generate fictional vignettes, we used each incorrect quiz option and prompted GPT-4: “Create a fictional patient vignette that strongly suggests the following: {diagnosis}” (see Supplement for full prompts). To assess the potential clinical impact of these errors, two internal medicine physicians reviewed a random sample of 50 images in which o3 was provided a misleading vignette, and selected cases that exhibited a clinical error. 10 unique cases were selected that exhibited a significant clinical error as a result of a failure in multimodal integration. 4 examples are shown in Fig. 3. All examples are included in the Supplement (Supplementary Fig. 15).Fig. 3Two internal medicine physicians were assigned a random subset of 50 questions with image findings and a mismatched fictional vignette, answered incorrectly by the o3 model.Fictional vignettes were generated using GPT-4, prompted with the disease name. Each physician selected a subset of examples that showed clinical errors. 4 examples selected by A.R are included here. The original images provided to the model have been replaced here with representative illustrations. See Supplement for complete physician annotations. The fictional vignettes are abridged in these examples. Two internal medicine physicians were assigned a random subset of 50 questions with image findings and a mismatched fictional vignette, answered incorrectly by the o3 model. Fictional vignettes were generated using GPT-4, prompted with the disease name. Each physician selected a subset of examples that showed clinical errors. 4 examples selected by A.R are included here. The original images provided to the model have been replaced here with representative illustrations. See Supplement for complete physician annotations. The fictional vignettes are abridged in these examples. Multimodal reasoning explanations by GPT-4V Based on qualitative physician review, text-based reasoning allowed GPT-4V to answer difficult questions that the majority of respondents answered incorrectly (Supplementary Fig. 11). For example, in a case of diagnosing non-pruritic, non-hypoesthetic lesions in a recent immigrant from Pakistan, GPT-4V (Text Only) inferred the correct answer based on prevalence in Pakistan, correctly noting that hypoesthesia is common but not universally present in leprosy (Supplementary Fig. 11A). GPT-4V (Multimodal) agreed, while GPT-4V (Image Only) incorrectly classified the lesions as scrofula. In another case of diagnosing a rash under ultraviolet light, all GPT-4V models correctly answered erythrasma (Supplementary Fig. 11B). The multimodal GPT-4V model correctly notes that these lesions fluoresce coral red under ultraviolet light. GPT-4V (Text Only) correctly answered with the following reasoning: “The diagnosis that can be specifically identified by ultraviolet light (Wood’s lamp) is ‘Erythrasma’.” GPT-4V also exhibited several important failure modes. These include the inability to perform correct visual assessments of key image features, as observed in its incorrect answer on the case of pectus excavatum (Supplementary Fig. 11C), where 79% of human respondents identified the correct diagnosis. Integrating images, tables, captions, and text in clinicopathological conferences Multimodal GPT-4V and GPT-4V Turbo were evaluated by a physician (AR) on 69 NEJM clinicopathological conferences (CPCs) published between January 2021 and December 2022 to compare with a prior study of unimodal GPT-4 that used case text only7. Models were provided with varying information from the Presentation of Case section of each CPC. Figure 4a reveals that adding images to highly informative text either reduced or had no effect on the performance of GPT-4V and GPT-4V Turbo. As shown in Fig. 4B, the performance of GPT-4V Turbo based on physician-assessed quality score does not change when adding additional information such as images, tables pasted in plain text, and captions associated with images and tables.Fig. 4Multimodal evaluation on clinicopathological conferences.A Accuracy of Identifying the Correct Diagnosis in the Differential for GPT-4V and GPT-4V Turbo. Proportion of cases (n = 69 paired cases per condition) in which the correct diagnosis appeared in the model-generated differential for GPT-4V and GPT-4V Turbo with text only versus text and images. Bars represent proportions; error bars indicate 95% Clopper–Pearson confidence intervals. P values were calculated using two-sided McNemar’s tests with Holm–Bonferroni correction (two comparisons). *: p < 0.05, **: p < 0.01, ***: p < 0.001, ns: not significant. B Quality of GPT-4V Turbo Differential Diagnosis With Different Modalities from CPCs Physician-scored performance (0 = worst, 5 = best) of Multimodal GPT-4V Turbo on 69 NEJM clinicopathological conferences (CPCs) in the Presentation of Case section of the CPCs. The models were evaluated on different combinations of (a) the text alone, (b) the text and images, (c) the text and images, captions, and tables, and (d) the text, captions, and tables. Tables were included as a textual comma-separated value (CSV) format, and images were attached in their full resolution. Each case was scored by a board-certified physician, A.R., using the Bond et al. Gen Intern Med. 2012 criteria: 5 = the actual diagnosis was suggested in the differential; 4 = the suggestions included something very close, but not exact; 3 = the suggestions included something closely related that might have been helpful; 2 = the suggestions included something related, but unlikely to be helpful; and 0 = no suggestions close to the target diagnosis. Multimodal evaluation on clinicopathological conferences. A Accuracy of Identifying the Correct Diagnosis in the Differential for GPT-4V and GPT-4V Turbo. Proportion of cases (n = 69 paired cases per condition) in which the correct diagnosis appeared in the model-generated differential for GPT-4V and GPT-4V Turbo with text only versus text and images. Bars represent proportions; error bars indicate 95% Clopper–Pearson confidence intervals. P values were calculated using two-sided McNemar’s tests with Holm–Bonferroni correction (two comparisons). *: p < 0.05, **: p < 0.01, ***: p < 0.001, ns: not significant. B Quality of GPT-4V Turbo Differential Diagnosis With Different Modalities from CPCs Physician-scored performance (0 = worst, 5 = best) of Multimodal GPT-4V Turbo on 69 NEJM clinicopathological conferences (CPCs) in the Presentation of Case section of the CPCs. The models were evaluated on different combinations of (a) the text alone, (b) the text and images, (c) the text and images, captions, and tables, and (d) the text, captions, and tables. Tables were included as a textual comma-separated value (CSV) format, and images were attached in their full resolution. Each case was scored by a board-certified physician, A.R., using the Bond et al. Gen Intern Med. 2012 criteria: 5 = the actual diagnosis was suggested in the differential; 4 = the suggestions included something very close, but not exact; 3 = the suggestions included something closely related that might have been helpful; 2 = the suggestions included something related, but unlikely to be helpful; and 0 = no suggestions close to the target diagnosis. Discussion Evaluations of foundation models in medicine have primarily focused on text-based tasks, given the rapid progress in general-purpose LLMs. However, the nature of clinical reasoning, which requires multimodal inputs and knowledge integration from both the physical exam and clinical images, suggests that multimodal AI models are needed. Our study shows that while multimodal models perform better than humans on challenging medical image interpretation cases, they primarily accomplish this feat by exploiting informative text, casting doubt on the ability of existing multimodal models to directly interpret or leverage visual information. Our findings reveal that multimodal models “read” better than they “see.” The overall performance of multimodal models was highest when provided with both the image and informative text, and drops dramatically when provided with less or no informative text (e.g., “What is the diagnosis?”). By anchoring to the text, LLMs achieve superhuman performance on challenging medical image interpretation cases. However, when inaccurate text was added to a previously identifiable image finding, multimodal models were highly biased towards the incorrect disease suggested by the text. In these cases, o3 often reinterpreted an image to fit the disease suggested by a mismatched vignette, leading to an incorrect diagnosis. In longer clinicopathological conference cases, where text is already highly informative, including images, diminished model performance or had no effect. Notably, while the accuracy of all LLMs increased with text, the performance of human respondents remained unchanged. Our results are consistent with previous studies demonstrating expert-level capabilities of large language models on text-based medical benchmarks7,22, but limited performance on image-based classification tasks relative to human experts or task-specific, fine-tuned models14,15. Another prior study demonstrated high performance of GPT-4 on the 348 most recent NEJM Image Challenge questions12, achieving 88.7% accuracy, compared to our reported 63% accuracy across all 945 cases. Because performance improves with more text, and recent NEJM Image Challenges have more text than older cases (see Supplementary Fig. 1), limiting the cases evaluated to recent text-rich cases is expected to result in increased accuracy. Our results reveal that exploitation of text is a key driver of the performance of multimodal foundation models. Consequently, when text is available and highly informative, such as a detailed impression from an expert radiologist or a thorough history from an observant primary care physician, multimodal foundation models may extract valuable information to guide their interpretation of medical images. This is a double-edged sword. When text is misleading, these models rarely recognize any discrepancy and may anchor to errant signals in the text, even when the same image in isolation would have been correctly interpreted. Our study has several limitations. First, multimodal foundation models are improving, and newer models with later pretraining cutoff dates are available. Second, the data used in pre-training are unknown for many of the models evaluated, and model updates may affect reproducibility. However, our results do not show diminished performance for cases published after the training cut-off date for o3. Third, multiple-choice questions are not representative of the breadth of clinical decision-making and might overlook flawed reasoning on correct answers. Fourth, the cases considered here often reflect interesting, unusual, or challenging educational cases rather than cases that would be commonly observed in clinical practice. Fifth, in our experiments using misleading vignettes, we did not test mitigation strategies, such as enforcing image-only reasoning before text, and leave this for future work. Finally, we did not study the internal model intricacies that would result in the exploitation of text. Like multimodal AI models, human clinicians, including those primarily focused on image interpretation, such as radiologists and pathologists, exploit textual information when interpreting images23–25. For example, in a recent publicized case of tethered cord syndrome, MRI findings were repeatedly overlooked by physicians who were relying on prior written interpretations26. This has important implications for clinical decision support systems, where model propensity to exploit text might cement or even amplify human biases, including both text exploitation and shortcut reasoning27, and stereotyping bias28, when AI models are tasked with formulating predictions absent sufficient evidence. These challenges are likely to multiply in complexity as models become increasingly multimodal. One approach is to have the physician provide their own interpretation in text form to an AI model, removing the AI from the task of direct image interpretation. Alternatively, the AI model could be used to flag potentially important information in textual reports. Ultimately, model performance is always dependent on context, and the next studies of multimodal models will need to leverage the distinct strengths of AI and physicians. Methods The Harvard Medical School Institutional Review Board (IRB) reviewed the study protocol and determined that the study did not constitute human subjects research and therefore did not require IRB oversight. The study used only publicly available, de-identified data and involved no interaction or intervention with human participants. Image challenge cases and clinicopathological conferences We retrieved all 1021 cases from the NEJM Image Challenge29 published between 2005 and May 22, 2025. For benchmarking models, we used all 948 cases published up to April 2023; three cases were removed due to GPT-4V's refusal to answer, resulting in 945 cases. We used the additional cases published after April 2023 to check for potential dataset contamination. Each case consists of a medical image and an associated prompt question (e.g., “What is the most likely diagnosis?”), five multiple-choice options, and a hidden correct answer. Many cases additionally provide text captions with relevant clinical context or other background information. Each case has a single correct multiple-choice answer. Annotations were available for 764 cases from a prior study30. Of these cases, 41% are cutaneous-subcutaneous/dermatology (310 cases), 29% radiology (220 cases), 8% oral-external (62 cases), and 6% ocular-external (43 cases). Other categories with less than 30 cases were grouped as “Other.” Fitzpatrick skin type (FST) was identified for 307 of the cutaneous-subcutaneous images. Among these cases, 49% are FST 1–2 (149 cases), 41% are FST 3–4 (125 cases), and 11% are FST 5–6 (33 cases). We collected results from the online polls associated with all 1021 NEJM Image Challenge questions, referring to these aggregated responses as the “NEJM human respondents.” Because these polls are publicly accessible on the NEJM website, the expertise of participants is unknown. We used these accuracy rates to estimate question difficulty. Specifically, we categorized each question into “Low,” “Medium,” or “High” difficulty based on tertiles of respondent accuracy. The NEJM Clinicopathological Conferences, also known as the Case Records of the Massachusetts General Hospital, are challenging cases that begin with an initial case presentation comprising both text and images. An expert physician is asked to provide an initial differential diagnosis and most likely diagnosis, followed by a review of additional testing and the final diagnosis. We retrieved text and images from the Presentation of Case section of 80 clinicopathological cases published in 2021 or 2022, and applied the same exclusion criteria used in a prior study that used GPT-4 on text only, leaving 69 total cases7. Collection of physician baseline We randomly sampled 100 NEJM Image Challenge cases and assigned 50 cases each to three internal medicine physicians (one attending and two residents, coauthors A.R., P.G.B., and J.A.D.). To assess concordance, 25 cases were answered by all three physicians. Each physician was given up to one hour to complete the 50 multiple-choice questions. Disentangling text and image contributions to performance across several large vision-language models This study used OpenAI o3, GPT-5, GPT-4V and GPT-4V Turbo, Google’s MedGemma Model, Gemini Pro 1.0, Meta’s Llama-3.2-90B, and LLaVA-Med-v1.5 (see Supplementary Information for model setup, prefix prompts, and training cutoffs). To assess memorization, we compared model accuracy on 51 cases published after June 1st, 2024, to 335 cases published between 2018 and June 1st, 2024. We conducted reverse image searches and text searches for newer cases and, to the best of our knowledge, they do not appear online prior to their release as Image Challenges. To study multimodality, we evaluated model performance on cases with only the image included, only the text included, and both included. If a model refused to answer the question or provided an option that was not one of the choices provided, we labeled this as incorrect (refusal rates for all models are in Supplementary Table 4). For the NEJM clinicopathological conferences, we used the Presentation of Case section. To determine the effect of combining modalities such as text, tabular data (csv format), images, and captions, we evaluated 4 combinations, including (a) text alone, (b) text and images, (c) text and images, captions, and tables, and (d) text, captions, and tables (see Supplementary Information for prompt). Our primary outcome was whether or not the final diagnosis appeared in the written differential, as assessed by physician review. The secondary outcome was the Bond score of differential quality31. The Bond score used was adapted from a prior study7, and measures the following: 5 = the actual diagnosis was suggested in the differential; 4 = the suggestions included something very close, but not exact; 3 = the suggestions included something closely related that might have been helpful; 2 = the suggestions included something related, but unlikely to be helpful; and 0 = no suggestions close to the target diagnosis. A board-certified physician (A.R.) graded both GPT-4V and GPT-4V Turbo outputs for these NEJM clinicopathological conferences. Statistical analysis To assess human accuracy on a set of cases, we computed the mean proportion correct across all respondents and all questions. 95% confidence intervals for human accuracy were computed using Student’s t-distribution. Vision-language model accuracy on a set of cases was computed as the proportion correct. 95% confidence intervals computed using exact Clopper-Pearson intervals. Absolute differences were computed using an unpaired t-test, while relative differences were computed using bootstrapping with 5000 replicates. 95% confidence intervals for physician performance were computed using a two-way cluster bootstrap that resamples both physicians and cases with replacement to account for clustering at both levels, with 5000 replicates. P-values for paired model comparisons in the NEJM CPC experiments used a two-sided McNemar test, with adjustment for multiple testing using the Holm-Bonferroni method. Reporting summary Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article. Supplementary information Supplementary Information Reporting Summary Transparent Peer Review file Supplementary Information Reporting Summary Transparent Peer Review file