Benchmarking reliability and calibration of LLMs for multi-cancer early detection test communication Authors: Koki Takabatake, Maria Sol Rosito, Danielle Braun, Catherine Marinac, Elizabeth O’Donnell, Giovanni Parmigiani Published: 2026 ABSTRACT AbstractObjectivesTo assess the reliability and confidence calibration of Large Language Models (LLMs) when communicating multi-cancer early detection (MCED) test characteristics in an expert-informed clinical evaluation.Materials and MethodsWe curated 19 papers regarding prominent MCED tests and trials. Using Google’s NotebookLM, we generated a question set that two MCED experts reviewed and finalized, comprising 94 multiple-choice questions (MCQs) and 13 free-response questions (FRQs). Five LLMs (ChatGPT, Claude, Gemini, Perplexity, OpenEvidence) each completed the question set three times. All models except OpenEvidence reported a confidence score (1-100) for every answer. We evaluated mean MCQ, FRQ, and overall accuracy. Confidence calibration for MCQs was assessed using reliability plots, Brier scores, and two high-confidence error metrics: proportion of answers with confidence ≥80 that were incorrect and proportion of incorrect answers with confidence ≥80. Between-model MCQ comparisons used Cochran’s Q and post hoc McNemar tests; FRQ comparisons used Friedman test.ResultsAll models demonstrated high overall accuracy. Gemini achieved the highest performance (96.5% MCQ, 96% FRQ, 96.3% overall), though comparisons involving Gemini should be interpreted cautiously as it was evaluated under different time and blinding conditions. MCQ performance differed significantly across models, while FRQ performance did not. Most errors involved recalling precise numerical estimates. Despite low Brier scores across models, several models made high-confidence errors.DiscussionSubdomain and calibration analyses revealed systematic weaknesses in numerical recall and confidence reliability despite strong accuracy.ConclusionLLMs may support communication of MCED information, but clinical use should incorporate verification workflows and human oversight.Keywords: Large language models, early detection of cancer, medical informatics, cancer screening, natural language processing Background and significance Multi-cancer early detection (MCED) tests are an emerging class of screening tools that can detect multiple cancers simultaneously often from a single, minimally invasive blood sample.1,2 These tools evaluate analytes, such as cell-free DNA (cfDNA) methylations and/or proteins to identify cancer signals and may return their predicted tissue of origin.3,4 MCEDs demonstrate high specificity for cancer detection, and scientists are optimistic that they could supplement current cancer-specific screening.2,4 However, even with high specificity, population-scale screening can generate false positives, overdiagnosis, and costly downstream cancer workups.4,5 In addition, many MCED tests prioritize specificity over sensitivity.6,7 Ongoing studies are being conducted to investigate where MCEDs add clinical value and how they should be implemented. As these technologies move toward clinical use, accurate and accessible explanations of their performance and limitations are increasingly important.8–10 Large language models (LLMs) could help meet this need by synthesizing and communicating complex scientific information, but they may also produce incomplete or false information.11,12 Currently, their reliability in accurately explaining MCED test characteristics has not been systematically evaluated. To address this gap, our study queried multiple LLMs with an expert-reviewed MCED question set, evaluated their responses for accuracy, and assessed calibration by investigating whether the models’ self-reported confidence in their answers matched their observed accuracy. We compared performance across LLMs overall, by question type, and by MCED topics. With these comparisons, the study aims to inform the feasibility of using LLMs as reliable tools for disseminating accurate, evidence-based information about MCED tests. Materials and methods Literature review and source curation We used a recent review by Guerra et al. (2024)4 as a scoping framework for source identification and screened its cited references to identify major MCED platforms, landmark studies, and recurring clinical themes relevant to question development. This ensured representation of key MCED topics, such as Galleri,13 CancerGuard,14 PanSeer,15 OneTest,16 and Trucheck.17 We then performed a targeted supplementary literature review to fill gaps and improve completeness of platform- and trial-specific coverage, including an overview article to provide better contextual coverage of PanSeer and Galleri as well as the National Cancer Institute report on Vanguard assay selection to capture the contemporary national MCED trial landscape.18,19 Sources were retained if they captured sufficient detail on assay mechanism, performance characteristics, trial design, and implementation considerations of MCED and were excluded if they were duplicative or insufficiently detailed for question generation. This process yielded 19 curated sources.1–5,18–31 spanning foundational MCED literature from 2018 through early 2025. The curated source set was approved by the MCED expert reviewers on our team for relevance and scope. After this review, we uploaded the 19 relevant MCED sources to Google’s NotebookLM. We then prompted NotebookLM to generate multiple choice (MCQ) and free response questions (FRQ) addressing key concepts of MCED test characteristics. We utilized Google’s NotebookLM because it could generate appropriate, citation-linked questions and summaries based on a curated set of papers.32–34 NotebookLM initially generated 100 MCQs and 20 FRQs along with a rubric, and we exported these outputs for expert review. Expert review and scoring rubric Our study team included an oncologist and a cancer epidemiologist with MCED expertise. We selected two MCED domain experts to provide subject-matter review of the draft question bank, answer set, and FRQ rubric for clinical accuracy, clarity, and relevance. We believe two experts were appropriate as our intent was to establish expert-informed reference standard for model evaluation rather than to conduct a formal content-validation study, for which a larger expert panel would be preferred. Experts provided comments directly on draft materials, the other members of the study team reviewed this feedback collectively, and revised materials were then returned to expert reviewers for confirmation. When additional concerns were identified, the materials underwent further revision and were recirculated until the experts considered the content acceptable. Questions that were ambiguous or of limited medical relevance were revised or removed. Our experts also reviewed FRQ scoring rubrics to ensure that awarded points reflected demonstration of clinically meaningful understanding by the LLMs. After iterative revisions, the final question bank consisted of 94 MCQs and 13 FRQs. Each MCQ question was scored as 1 point (maximum 94 points), and each FRQ had a maximum score of 1, 2, or 3 points. Across the 13 FRQs, the total possible FRQ score was 25 points. When combined, the question set yielded a total possible score of 119 points (see Supplementary Material S1). Evaluation We evaluated five different LLMs using this approved question set: ChatGPT 5.1 Thinking, Claude (Sonnet 4.5), Gemini 3.1 Pro, Perplexity, and OpenEvidence. All models were accessed through their native web interfaces in Google Chrome, rather than through a unified API environment. Generation settings, such as temperature, top-p, and other decoding parameters were not user-adjustable in these interfaces, and default LLM settings were thus used. Each model was queried with the full question set three independent times (“runs”) following the overall workflow shown in Figure 1. ChatGPT, Claude, Perplexity and OpenEvidence were queried between December 7 and 13, 2025. Gemini 3.1 Pro was added later to our analysis and was queried on March 26, 2026. For ChatGPT, Claude, Gemini, and Perplexity, we uploaded the question set through the web interface and used the same prompt template for each run: “Please answer all questions to the best of your ability.” This prompt elicited both MCQ and FRQ answers. After each model provided answers, we issued a follow-up prompt in the same conversation in all the models: “Please rate your confidence level from 1-100 in each of your answers. 100 is 100% confident, 1 is not confident at all.” Workflow for benchmarking large language models on MCED questions. Flow diagram shows the study pipeline. MCED papers were used to generate questions via NotebookLM, which were then reviewed by experts. Each LLM completed the question 3 times with confidence ratings. OpenEvidence required a modified workflow because the interface imposed word limits on user prompts and did not permit upload of question set. Thus, MCQs were submitted in small batches of roughly 5-8 questions at a time within the same session. Because OpenEvidence did not provide letter answers to MCQs with the prompt used for other LLMs, we replaced the prompt with “Please provide letter answers” after pasting each MCQ batch, all within the same inquiry. For OpenEvidence, FRQs were submitted one question at a time within the same session after completion of the MCQ section until all FRQ questions were answered. When OpenEvidence was prompted to provide confidence levels of its answers applying the same prompt used on other models, OpenEvidence was unable to complete this task, reporting that such inquiry was out of their scope. Each run was conducted in a new session to minimize carryover from prior context. We did not provide external reference documents beyond the question itself. All evaluated LLMs were permitted to access the open web during answering. Of note, only OpenEvidence routinely returned explicit citations to support its answers, whereas the other LLMs did not provide citations in their output. Accordingly, our evaluation reflects web-enabled LLM performance under typical user-facing conditions, but source traceability differed by model. Observed performance represents a combination of the underlying model’s generative capabilities and the model’s retrieval and source-integration infrastructure, rather than solely intrinsic parametric knowledge. We calculated the average score across these runs. All runs were scored against the expert-reviewed answer key. For MCQs, models provided either a single letter answer or a single letter answer with a brief explanation. MCQ responses were scored against the answer key based on the identified letter answer, rather than the brief explanation. Models did not produce multiple answer choices or ambiguous final selections. For FRQs, two study investigators distinct from the expert reviewers scored them independently using the expert-reviewed rubric. One scorer was blinded to model identity and the other was not. Gemini 3.1 Pro was added later to the study, and its FRQ responses were thus not scored under the same blinded conditions, as the additional set of responses were known to be from Gemini. Each required concept was worth 1 point. After independent scoring, the graders reviewed discrepancies together and attempted to reach consensus through discussion and clarification of rubric interpretation. When disagreement persisted and consensus could not be reached, 0.5 points were awarded to that specific concept. These disagreements most often occurred when one reviewer judged a response to have fully addressed a concept and the other judged it to be incomplete or insufficiently clear. Final FRQ scores reflected this adjudication process. Because each model answered the same questions and MCQ outcomes were binary (i.e, correct vs incorrect), between-model MCQ performance was compared via Cochran’s Q test, treating each question-run combination as a matched block. When the overall test was significant, post hoc pairwise McNemar tests with Holm correction were performed. For FRQs, because responses were graded with partial-credit rubrics and item totals varied, we compared models using the Friedman test on question-level proportional scores, defined as score earned divided by the maximum possible score for that item. Confidence ratings and calibration Confidence ratings were rescaled from 1-100 to a 0-1 range and interpreted as each model’s self-reported probability that its answer was correct. We summarized confidence variability by looking at its standard deviation and interquartile range to characterize how much each model varied its stated probability. We calibrated model performance, treating each MCQ as an independent prediction and pairing their confidence value with correctness (ie, binary value of 0 being incorrect and 1 being correct). Then, we constructed reliability plots of observed accuracy vs mean reported confidence with a 45° reference line. Points below the 45° line indicate overconfidence (reported confidence exceeds observed accuracy), whereas points above the line indicate underconfidence. Overall calibration for MCQs was summarized by the Brier score, defined as the mean squared error between the reported probability and correct answers. To assess performance, we also computed two additional metrics. One metric was Error Rate Among High Confidence Answers (ERHCA), defined as the proportion of answers with reported confidence ≥80 that were incorrect (P[incorrect | confidence ≥80]). The second metric was High Confidence Rate Among Errors (HCRE), defined as the proportion of incorrect answers for which the model reported confidence ≥80 (P[confidence ≥80 | incorrect]). These metrics quantified moments when models provided incorrect information while reporting strong internal certainty. Subdomain performance To analyze LLMs’ areas of strength in explaining MCED, we categorized MCQs into subdomains based on the primary topic of each question: MCED general concepts, Galleri/GRAIL MCED, PATHFINDER study, NHS-Galleri trial, CancerGuard test, PanSeer test, Trucheck test, OneTest, and Implementation and MCED value (see Supplementary Material S1). The MCED general concepts section captured foundational principles of these technologies, such as intended use and general mechanisms. Galleri/GRAIL MCED, PATHFINDER, NHS-Galleri trial, CancerGuard test, PanSeer test, Trucheck test, and OneTest sections focused on testing details of the MCED technology, performance metrics, study design, and pertinent findings. The Implementation and MCED Value encompassed questions pertaining to clinical integration, such as inquiries about benefits and harms, cost-effectiveness, and equity considerations. Within each subdomain, we calculated the mean number of correct responses across the three runs and derived corresponding percent-correct values. Results Overall performance across models Across all models and runs, LLMs demonstrated strong performance on both MCQ and FRQ sections (Figure 2). For the MCQ section, Gemini achieved the highest average accuracy [µ = 96.5%, standard deviation (SD) = 2.7%], followed by ChatGPT (µ = 92.9%, SD = 1.6%), Claude (µ = 92.6%, SD = 1.1%), OpenEvidence (µ = 90.1%, SD = 0.6%), and Perplexity (µ = 89.7%, SD = 3.7%). On paired item-level comparison, MCQ performance differed significantly across models overall (Cochran’s Q = 18.08, df = 4, P = .001). In pairwise McNemar analyses with Holm correction, Gemini significantly outperformed OpenEvidence (adjusted P = .005) and Perplexity (adjusted P = .005). No other pairwise MCQ comparisons were significant. For the FRQ section, Gemini achieved the highest mean accuracy (SD) of 96.0% (4.0%) across runs. ChatGPT, Claude, OpenEvidence, and Perplexity achieved high mean FRQ accuracies of 92.0% (2.0%), 92.7% (5.0%), 92.7% (6.1), and 91.3% (3.1%), respectively. On paired comparison of FRQ scores, there was no evidence of an overall significant difference between models [Friedman χ2(4)= 0.69, P = .95]. Because Gemini was evaluated later than the other models and its FRQ responses were not scored under the same blinded conditions, these between-model comparisons should be interpreted cautiously. Average Accuracy, defined as proportion of correct answers, of each LLM on multiple choice question (MCQ), free response question (FRQ), and overall, averaged across three independent runs. Bar chart compares mean accuracy across LLMs on MCQs, FRQs, and overall performance averaged across three independent runs. X-axis lists LLMs, and y-axis shows proportion correct. Comparative testing showed that MCQ performance differed overall across models, notably with Gemini significantly outperforming OpenEvidence and Perplexity (adjusted P = .005 for both). FRQ performance did not show statistically significant between-model differences. When combining MCQ and FRQ performance, Gemini achieved the highest overall accuracy (SD) at 96.4% (1.9%) across the three runs. ChatGPT, Claude, OpenEvidence, and Perplexity followed closely with overall accuracies of 92.7% (0.9%), 92.2% (1.2%), 90.6% (1.0%), and 90.1% (2.5%), respectively. Subdomain MCQ performance Average MCQ performance for each model across topic-specific subdomains is shown in Table 1. For MCED general concepts (9 items), all models achieved perfect accuracy, with reported mean scores of 9/9 (100%). Table 1.Average performance of each LLM by MCQa subdomain, # correct (%).Topics (N = number of questions)ChatGPTClaudeGeminiOpenEvidencePerplexityMCED General Concepts (N = 9)9 (100.0)9 (100.0)9 (100.0)9 (100.0)9 (100.0)Galleri/GRAIL MCED (N = 10)10 (100.0)10 (100.0)10 (100.0)9 (90.0)9 (90.0)PATHFINDER Study (N = 7)6.3 (90.5)5 (71.4)6.7 (95.7)5.3 (76.2)6 (85.7)NHS-Galleri Trial (N = 10)9.6 (96.7)10 (100.0)10 (100.0)10 (100.0)10 (100.0)CancerGuard Test (N = 11)10 (90.9)9.7 (87.9)10 (90.9)9.7 (87.9)8.7 (78.8)PanSeer Test (N = 10)10 (100.0)9 (90.0)9.7 (96.7)10 (100.0)9.7 (96.7)Trucheck (N = 10)7.3 (73.3)8.7 (86.7)9.7 (96.7)8.7 (86.7)8 (80.0)OneTest (N = 9)7.3 (81.5)8 (88.9)7.7 (76.7)5.3 (59.3)7 (77.8)Implementation and MCED value (N = 18)17.3 (96.3)17.7 (98.1)17.7 (98.1)16.7 (92.6)17 (94.4)aMultiple choice question. Average performance of each LLM by MCQa subdomain, # correct (%). Multiple choice question. Performance across individual MCED tests on other subdomains varied across models. For Galleri/GRAIL MCED (10 items), ChatGPT, Claude, and Gemini all scored 10 (100%), while OpenEvidence and Perplexity each averaged 9 (90%). In the CancerGuard subdomain (11 items), mean scores were 10 (90.9%) for ChatGPT and Gemini, 9.7 (87.9%) for Claude and OpenEvidence, and 8.7 (78.8%) for Perplexity. For PanSeer (10 items), ChatGPT and OpenEvidence both averaged 10 (100%), Claude 9 (90.0%), and Gemini and Perplexity 9.7 (96.7%) each. For Trucheck (10 items), mean correct answers were 7.3 (73.3%) for ChatGPT, 8.7 (86.7%) for Claude and OpenEvidence, 9.7 (96.7%) for Gemini, and 8 (80.0%) for Perplexity. For OneTest (9 items), ChatGPT averaged 7.3 (81.5%), Claude 8 (88.9%), Gemini 7.7 (76.7%), OpenEvidence 5.3 (59.3%), and Perplexity 7 (77.8%). Results for trial and implementation subdomains also showed variations. For the PATHFINDER study (7 items), mean correct responses were 6.3 (90.5%) for ChatGPT, 5 (71.4%) for Claude, 6.7 (95.7%) for Gemini, 5.3 (76.2%) for OpenEvidence, and 6 (85.7%) for Perplexity. For the NHS-Galleri trial (10 items), accuracy was high across all models, with mean scores of 9.6 (96.7%) for ChatGPT and 10 (100%) for Claude, Gemini, OpenEvidence, and Perplexity. In the Implementation and MCED value subdomain (18 items), mean scores were 17.3 (96.3%) for ChatGPT, 17.7 (98.1%) for Claude and Gemini, 16.7 (92.6%) for OpenEvidence, and 17 (94.4%) for Perplexity. After tabulating MCQs missed across three independent runs, we observed that most errors clustered around questions related to quantitative MCED testing performance metrics (sensitivity, specificity, PPV) compared to questions about trial endpoints and implementation questions (Table 2). There were 6 sensitivity-related questions in the question bank, and with 3 runs, each model had 18 opportunities to answer these questions. Across questions related to sensitivity of MCED tests, ChatGPT and Claude each missed 5 (27.8%), Gemini 3 (16.7%), OpenEvidence 9 (50.0%), and Perplexity 11 (61.1%). For questions related to specificity of MCED tests (5 questions; 15 opportunities), errors were more frequent overall: ChatGPT missed 10 opportunities (66.7%), OpenEvidence 9 (60.0%), Perplexity 7 (46.7%), Claude 6 (40.0%), and Gemini 4 (26.7%). PPV errors were concentrated in OpenEvidence and Perplexity (each 3/6; 50.0%), while ChatGPT, Claude, and Gemini were correct across all questions related to PPV of MCED tests (missed 0/6 each). Table 2.Missed MCQsa by LLM across MCED performance, trial endpoint, and implementation topics, # correct (%).Missed Topics (N = number of MCQs)Opportunities (N × 3 runs)ChatGPTClaudeGeminiOpenEvidencePerplexitySensitivity of MCED Tests (N = 6)185 (27.8)5 (27.8)3 (16.7)9 (50.0)11 (61.1)Specificity of MCED Tests (N = 5)1510 (66.7)6 (40.0)4 (26.7)9 (60.0)7 (46.7)PPV of MCED Tests (N = 2)60 (0)0 (0)0 (0)3 (50.0)3 (50.0)MCED Trial Endpointsb (N = 6)182 (11.1)6 (33.3)2 (11.1)5 (27.8)4 (22.2)MCED Implementationc (N = 2)62 (33.3)1 (16.7)1 (16.7)2 (33.3)3 (50.0)aNumber of incorrect answers out of 94 multiple choice questions across 3 runs.bTested the following concepts: % of true positives that achieved diagnostic resolution; # of serious adverse events; accuracy of predicted cancer signal origin; # of common cancers detected, concordant rate, Trucheck overall accuracy (see Supplementary Material S1).cIncluded following concepts: potential value-based price of an MCED test at a willingness-to-pay threshold of $100 000 per quality-adjusted life-year gained in the US; Number Needed to Screen (NNS) to detect one cancer using the MCED test in PATHFINDER (see Supplementary Material S1). Missed MCQsa by LLM across MCED performance, trial endpoint, and implementation topics, # correct (%). Number of incorrect answers out of 94 multiple choice questions across 3 runs. Tested the following concepts: % of true positives that achieved diagnostic resolution; # of serious adverse events; accuracy of predicted cancer signal origin; # of common cancers detected, concordant rate, Trucheck overall accuracy (see Supplementary Material S1). Included following concepts: potential value-based price of an MCED test at a willingness-to-pay threshold of $100 000 per quality-adjusted life-year gained in the US; Number Needed to Screen (NNS) to detect one cancer using the MCED test in PATHFINDER (see Supplementary Material S1). Trial endpoint questions (6 questions; 18 opportunities) showed fewer but still notable errors: ChatGPT and Gemini each missed 2 opportunities (11.1%), Claude 6 (33.3%), OpenEvidence 5 (27.8%), and Perplexity 4 (22.2%). Finally, implementation-focused questions (2 questions; 6 opportunities) were missed by multiple models, with ChatGPT and OpenEvidence each incorrect on 2 opportunities (33.3%), Claude and Gemini each missed 1 (16.7%), Perplexity 3 (50.0%). Similar to performance metric questions, trial end point and implementation questions required numerical recall, such as percentage of true positives that achieve diagnostic resolution, the number needed to screen to detect one cancer with MCED, concordant rate, etc. None of the MCQ errors were from qualitative questions that required understanding of mechanism, risks and benefits, etc. Thus, errors were disproportionately concentrated in quantitative recall about MCED performance and characteristics rather than inquiries about concept-mechanisms items. Calibration, ERHCA, and HCRE Calibration was evaluated by grouping responses into bins of self-reported confidence and plotting mean observed accuracy against the mean confidence for each bin (Figure 3, Table 3). For ChatGPT, confidence values ranged from approximately 0.65 to 0.96 (mean = 0.84, SD = 0.06), with corresponding observed accuracies ranging from roughly 0.66 to 1.00. Across bins, ChatGPT showed mixed calibration: multiple mid-to-high confidence bins (0.75-0.90) fell above the 45° line (underconfidence), while at one lower confidence bin (0.70) fell below the line (overconfidence). Claude’s points were concentrated in higher confidence bins (mean = 0.96, SD = 0.04), with observed accuracies largely between about 0.89 and 1.00. Within this narrower confidence range, Claude’s bins included points above the line (underconfidence) at some confidence levels (0.80 to ∼0.91) and points below the line (overconfidence) at the upper end of the confidence range (∼0.96-1.00). Gemini’s points were tightly clustered at confidence level of 1 (SD = 0), preventing meaningful calibration analyses. Perplexity’s confidence bins spanned roughly 0.80-0.97 (mean = 0.97, SD = 0.03), with observed accuracies ranging from about 0.75 to just under 1.00; several bins in the 0.90-0.96 confidence range had observed accuracies below 0.90, indicating overconfidence in this range. A 45° reference line is shown to illustrate exact agreement between confidence and accuracy across the range of values. Reliability plots: calibration of LLM Confidence vs Accuracy. Observed accuracy is plotted against mean self-reported confidence, with responses grouped into confidence bins. The range of confidence bins on x-axis was truncated, as no models reported confidence level below 0.65. Each point represents the average accuracy within a confidence bin across all questions and runs for a given model. The 45° reference line in gray indicates perfect agreement between confidence and observed accuracy. Gemini was not plotted given its limited heterogeneity, as it indicated 100% confidence in all inquiries. Table 3.Brier Scorea, ERHCAb, HCREc for MCQd by LLMe.Brier ScoreHCREERHCAMean confidence (SD)Interquartile range (25th-75th)ChatGPT0.070.650.050.84 (0.06)0.80-0.90Claude0.071.000.070.96 (0.04)0.95-1.00Gemini0.041.000.041.00 (0)1.00-1.00Perplexity0.101.000.100.97 (0.03)0.95-0.99aBrier score is defined as the mean squared error between the reported probability and correctness.bError Rate Among High Confidence Answers (ERHCA) is defined as the proportion of answers with reported confidence ≥80 that were incorrect (P[incorrect | confidence ≥80]).cHigh Confidence Rate Among Errors (HCRE), defined as the proportion of incorrect answers for which the model reported confidence ≥80 (P[confidence ≥80 | incorrect]).dMultiple Choice Questions.eOpenEvidence did not provide confidence scores and was excluded from calibration analyses. Brier Scorea, ERHCAb, HCREc for MCQd by LLMe. Brier score is defined as the mean squared error between the reported probability and correctness. Error Rate Among High Confidence Answers (ERHCA) is defined as the proportion of answers with reported confidence ≥80 that were incorrect (P[incorrect | confidence ≥80]). High Confidence Rate Among Errors (HCRE), defined as the proportion of incorrect answers for which the model reported confidence ≥80 (P[confidence ≥80 | incorrect]). Multiple Choice Questions. OpenEvidence did not provide confidence scores and was excluded from calibration analyses. Summary calibration metrics (Table 3) showed low Brier scores across models, ranging from 0.04 to 0.10 across LLMs: Gemini had the lowest observed Brier score of 0.04, ChatGPT and Claude each had Brier scores of 0.07, and Perplexity had a Brier score of 0.10. However, Gemini reported uniformly maximal confidence (mean 1.00, SD 0), which prevented meaningful calibration assessment across confidence levels. Gemini’s low Brier score reflects high overall accuracy under constant stated certainty rather than informative discrimination of uncertainty. Thus, Gemini’s calibration results should be interpreted cautiously, as they do not reflect meaningful heterogeneity in reported confidence. By contrast, ChatGPT exhibited the broadest confidence range (mean 0.84, SD 0.06), enabling a more informative reliability curve with a generally monotonic increase in accuracy as confidence increased (Figure 3). Perplexity showed evidence of high-confidence overstatement, with several bins in the 0.90-0.96 range falling below the 45° line. Discussion In this study, we developed and utilized a clinical expert-reviewed MCED question bank to benchmark the performance of several LLMs about MCED-related knowledge. Across models, all five LLMs achieved high overall accuracy with modest run-to-run variability on both MCQs and FRQs. Though Gemini achieved the highest scores, this finding should be interpreted cautiously because it was evaluated later than the other models and under partially different conditions. Despite these strong aggregate scores, further analyses provided insight to systematic weaknesses in reporting accurate quantitative data and important differences in how models represented their own confidence. Overall and subdomain performance All evaluated LLMs achieved high overall accuracy on the MCED question bank. Subdomain analyses showed perfect performance for general MCED concepts, Galleri/GRAIL questions, and NHS-Galleri trial items for most models, indicating that LLMs have strong abilities to accurately capture foundational and the most extensively studied aspects of MCED.3,5,29 This pattern aligns with studies showing strong LLM performance on established cancer topics and guidelines.35–37 Performance was more variable for questions on Trucheck, OneTest, CancerGuard, and PATHFINDER, suggesting consistent understanding of MCEDs with more limited peer-reviewed evidence.38,39 Formal paired comparison supported an overall significant difference in MCQ performance across models, and post hoc analyses highlighted that Gemini significantly outperformed OpenEvidence and Perplexity. These findings suggest that although Gemini showed the strongest structured-recall performance, evidence for between-model differences was limited mainly to a few LLMs with lower-performing MCQ capabilities rather than across the top-performing group. FRQ performance provided additional insight into LLM reliability in explaining MCED. Mean FRQ accuracy was high across models, with no significant overall between-model differences. Our findings suggest greater between-model separation for structured recall than for rubric-based FRQ explanation. The error table (Table 2) highlights that most missed items clustered around numerical metrics such as sensitivity, specificity, PPV, and other trial- and implementation-related quantities. These findings suggest that models are more likely to perform less accurately when precise quantitative recall is required than when questions focus on qualitative mechanisms or conceptual understanding. Calibration and confidence Calibration analyses provide a complementary perspective on LLM behavior beyond overall accuracy. ChatGPT, Claude, Gemini, and Perplexity showed low Brier scores, but interpretation depended on distribution of reported confidence values. Though Gemini had the lowest Brier score, it reported 100% confidence for all responses, preventing meaningful calibration assessment across confidence levels. Its calibration metric thus reflects high overall accuracy under constant certainty rather than informative uncertainty discrimination. By contrast, ChatGPT showed the broadest spread of confidence values and was mostly underconfident between 0.75-0.95 confidence bins, highlighting a higher level of caution when answering MCED questions compared to other LLMs. Perplexity showed relatively marked overconfidence in the 0.90-0.95 confidence range, where observed accuracy fell well below self-reported confidence. The high-confidence error metrics supported these visual findings. ERHCA remained low or modestly low across models, indicating that only a minority of high-confidence answers were incorrect. However, HCRE was high for Claude and Perplexity, showing that when these models made mistakes, they often did so with high confidence. Overall, the confidence metrics and reliability plots underscore that high self-reported confidence should not substitute for verification.40–43 Conclusion The strong performance on concept- and implementation-focused questions suggests that LLMs may be useful tools for communicating MCED principles to users. They appear well suited for summarizing trial designs, explaining the mechanism behind MCEDs, and outlining general benefits and limitations. These roles align with studies showing that LLMs can produce largely accurate, comprehensible summaries about oncology that are rated as useful by physicians and patients.35,37 However, errors clustered around numerical metrics and health-economic quantities, supporting the need for external verification when these values are communicated. This finding also aligns with broader findings that LLMs can misrepresent numerical values even when their qualitative reasoning is sound.44,45 Additionally, the calibration results emphasize that model-reported confidence should not be substituted for accuracy. Even models with good calibration metrics produced incorrect answers at high confidence. Therefore, it may be preferable to design interfaces that combine confidence scores with citation requirements or automated checks against known reference values when supported reasoning is sparse or ambiguous. Limitations Several limitations should be acknowledged. The question bank was derived from 19 papers and therefore reflects the scope, emphasis, and time frame of that literature. Moreover, the question bank was reviewed by two subject-matter experts, and a larger and more diverse expert panel may have strengthened content review. Additionally, FRQ scoring, although guided by structured rubrics including one blinded reviewer, retained some subjectivity, and the final discrepancy-consensus process could still permit residual bias. Also, Gemini 3.1 Pro answered the question bank on March 2026, rather than December 2025 like the other LLMs, and its FRQ scoring could not be conducted under blinded conditions. This dual limitation with Gemini 3.1 Pro contributes to some residual comparability bias. Furthermore, our experiment used structured expert-reviewed prompts rather than naturalistic user interactions, so the findings should be interpreted as reflecting model behavior under controlled benchmarking conditions rather than real-world use. Nevertheless, some behaviors observed in this benchmark, particularly overconfidence and uncertainty calibration, may still be relevant in more naturalistic settings, even if their magnitude differs. Additionally, because all models were evaluated with open-web access enabled, observed differences may reflect platform-level retrieval and source-integration differences in addition to underlying model behavior. Finally, OpenEvidence’s lack of confidence ratings and Gemini’s constant 100% confidence limited our calibration analysis for these LLMs. Future directions Future work should extend this framework to more natural user questions, including patient-facing and clinician-facing prompts that mix MCED information with individual risk factors, comorbidities, or concurrent screening tests. Incorporating retrieval from updated trial registries and clinical guidelines may help reduce numerical errors. Finally, future studies should evaluate how well LLMs assist in navigating care after a positive MCED result, such as explaining downstream diagnostic workups and referrals. Supplementary Material Contributor Information Koki Takabatake, Boston University Chobanian and Avedisian School of Medicine, Boston, MA, 02118, United States; Department of Data Sciences, Dana-Farber Cancer Institute, Boston, MA, 02215, United States. Maria Sol Rosito, Department of Data Sciences, Dana-Farber Cancer Institute, Boston, MA, 02215, United States; Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, 02115, United States. Danielle Braun, Department of Data Sciences, Dana-Farber Cancer Institute, Boston, MA, 02215, United States; Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, 02115, United States. Catherine Marinac, Department of Medical Oncology, Dana-Farber Cancer Institute, Boston, MA, 02215, United States. Elizabeth O’Donnell, Department of Medical Oncology, Dana-Farber Cancer Institute, Boston, MA, 02215, United States. Giovanni Parmigiani, Department of Data Sciences, Dana-Farber Cancer Institute, Boston, MA, 02215, United States; Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, 02115, United States. Author contributions Koki Takabatake (Formal analysis, Investigation, Methodology, Writing—original draft, Writing—review & editing), Maria Sol Rosito (Investigation, Supervision, Validation, Writing—review & editing), Danielle Braun (Methodology, Supervision, Writing—review & editing), Catherine R. Marinac (Methodology, Validation, Writing—review & editing), Elizabeth O’Donnell (Methodology, Validation, Writing—review & editing), and Giovanni Parmigiani (Methodology, Project administration, Validation, Writing—review & editing) Supplementary material Supplementary material is available at JAMIA Open online. Conflicts of interest D.B. and G.P. colead the BayesMendel Laboratory, which develops and maintains the BayesMendel and Fam3PRO software package. BayesMendel includes a variety of risk assessment tools, including BRCAPRO, PancPRO, MelaPRO, MMRpro, and Fam3PRO, which are licensed for commercial use. None of the licensing fees for BayesMendel generate personal income for laboratory members. While G.P. does not consider any of these activities to be in conflict with the research in this committee, G.P. wishes to disclose that he holds equity in Phaeno Biotechnologies and currently consults for Delphi Diagnostics. E.O. has served as a consultant to Natera, Grail, Exact Sciences, Pfizer, Sanofi, and Bristol Myers Squibb, and has received research funding from Regeneron. C.M. serves on the Exact Sciences Corporation Advisory Board and has received honoraria from Exact Sciences Corporation and consulting fees from Natera. The remaining authors declare no competing interests. Funding G.P. would like to acknowledge generous donations from the Lavine and Dobson families. Data availability The data underlying this article are available in the Dryad Digital Repository at https://doi.org/10.5061/dryad.zs7h44jrt. References Associated Data Supplementary Materials Data Availability Statement The data underlying this article are available in the Dryad Digital Repository at https://doi.org/10.5061/dryad.zs7h44jrt.