Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations Authors: Christopher R Stephenson, Jithinraj Edakkanambeth Varayil, Bradley R Salonen, Elizabeth C Wight, Phil D Sang, Aditya K Ghosh, Dennis L Shung, Ivana T Croghan, Darrell R Schroeder, Ryan T Hurt Published: 2026 ABSTRACT AbstractObjectiveTo evaluate whether a context-specific large language model, OpenEvidence, can generate gastroenterology/hepatology eConsult responses for metabolic dysfunction-associated steatotic liver disease (MASLD), while characterizing the frequency and potential severity of errors.Patients and MethodsIn this retrospective cohort study at a tertiary academic medical center, 125 specialist electronic consultations (eConsults) for MASLD (from January 2020 to December 2024) were paired with OpenEvidence-generated eConsults produced using OpenEvidence and standardized prompts. Five blinded physician reviewers evaluated all eConsults using a previously validated instrument. A board-certified gastroenterologist independently reviewed OpenEvidence-generated eConsults to identify errors and assess potential for harm.ResultsOpenEvidence-generated eConsults received higher overall eConsult Specialist Quality of Response scores than specialist eConsults (mean difference, 6.4; 95% CI, 6.0-6.9; P<.001) and higher global quality ratings (odds ratio [OR], 16.1; 95% CI, 12.0-21.5; P<.001). OpenEvidence-generated eConsults were more likely to be rated as providing an appropriate amount of information (70.9% vs 33.9%; OR, 6.4; 95% CI, 4.8-8.5; P<.001). Of 125 OpenEvidence-generated eConsults, 12 contained minor errors and 1 contained a potentially major error.ConclusionIn this blind comparison of paired eConsults for MASLD, OpenEvidence-generated responses were rated higher in perceived quality and informational adequacy than specialist-authored responses, but occasionally clinically relevant errors occurred. These findings support further evaluation of OpenEvidence-assisted eConsult workflows for protocol-driven conditions. There is a growing shortage of specialty physicians across the United States, threatening timely access to expert care for patients and specialty expertise for primary care clinicians (PCCs).1 National workforce projections estimate a potential shortfall of up to 180,000 physicians by 2037, although realized shortages vary by specialty, geography, and health care system context.2,3 Additionally patient care continues to grow more complex, leading to increasing referral wait times and decreasing access to in-person subspecialty care. One solution to this shortfall has been electronic consultations (eConsults).4 eConsults enable asynchronous communication between PCCs and specialists without in-person visits.5 eConsults were implemented to improve access to specialty care, reduce unnecessary referrals, and support PCCs in managing increasingly complex patients. eConsults have been increasingly integrated into various health care systems, including Mayo Clinic, Kaiser Permanente, and the Veterans Health Administration.6 They increase access to specialty care in rural and underserved areas, leading to faster specialist access and improved satisfaction among PCCs.7 eConsults can also lead to decreased health care utilization and cost savings.8,9 However despite their use in streamlining care, specialty expertise is still a limited and finite resource. Large language models (LLMs) like OpenAI’s GPT-5 and knowledge-specific systems, such as OpenEvidence, have reported the ability to generate patient-specific, evidence-based recommendations.10 Artificial intelligence (AI) tools have already reported promise for clinical reasoning and decision support, and unlike specialty physician expertise, LLMs are scalable.11 OpenEvidence, a context-specific LLM for medical applications, can provide clinical decision support and synthesize peer-reviewed literature and clinical guidelines for health care clinicians.12 These responses are structured and often include differential diagnoses, treatment recommendations, and references to guidelines. Recommendations are cited and pulled from peer-reviewed literature. Previous studies have shown that LLM responses may receive higher ratings than physician responses for patient-facing communication, although length, organization, and perceived empathy may have influenced reviewer preferences.13 In addition, LLMs have produced favorable output. One study reported how specialists preferred LLM responses compared with generalist physician responses in a pilot study using real-world medical questions.10 More specialized LLM tools, such as OpenEvidence, have reported high accuracy and performance on the real clinical questions submitted by physicians.14 Another study compared OpenEvidence responses to real clinical dilemmas, including a blinded comparison with human-generated responses, and found comparable subjective ratings of reviewer satisfaction.15 These studies have reported that LLM-generated responses may be perceived favorably, but they have not evaluated a large set of paired, real-world specialist eConsults using an eConsult-specific quality instrument. The present study was therefore designed to use real-world eConsults to preserve the variability, incomplete information, and practical constraints encountered in physician-to-specialist communication. Similarly, the eConsult Specialist Quality of Response (eSQUARE) instrument was selected to assess general, management, and communication quality, which is more specific to the eConsult task than global ratings of preference, utility, or satisfaction. A separate review for clinically relevant errors also provides information that cannot be inferred from perceived quality scores alone. Although eConsults improve access to specialty expertise, they may also introduce risk when recommendations are applied without a full clinical context. These risks may be amplified in AI-generated responses because LLMs must infer clinical nuance from limited input and may struggle with incomplete data.16 Overly aggressive management advice may lead to unnecessary testing and overtreatment, particularly in patients with low pretest probability of disease. Conversely, incomplete recognition of atypical presentations may delay appropriate referral. While widely available LLMs, such as ChatGPT, may be able to provide clinical decision support for eConsults, there are concerns regarding accuracy, hallucinations, biases, and patient privacy concerns.17, 18, 19 Data regarding the performance and safety of context-specific platforms such as OpenEvidence within eConsult workflows remain limited. Gastroenterology and hepatology (GIH) represents an ideal specialty to evaluate a context-specific LLM’s eConsultation ability because of the specialty’s high consultation volumes and the growing prevalence of complex chronic conditions. Within GIH, metabolic dysfunction-associated steatotic liver disease (MASLD) serves as an ideal use case for context-specific AI tools for PCCs. MASLD impacts an estimated 25%-35% of adults worldwide, and because most patients with MASLD are managed by PCCs or general internists, the condition represents a common source of eConsult requests.20,21 These consultations often center around risk stratification, diagnostic workup, and management of underlying metabolic conditions.22 In addition, GIH consultation may be limited in resource-constrained environments, such as rural health. The high volume of MASLD-related consults and decreasing GIH access provide an ideal opportunity to evaluate AI-generated eConsults, produced by OpenEvidence. We therefore compared OpenEvidence-generated and gastroenterologist-authored responses to 125 matched MASLD eConsult cases using 5 blinded physician reviewers. We hypothesized that OpenEvidence-generated eConsults would report comparable or higher perceived quality, whereas separately characterizing the frequency and potential severity of clinically relevant errors. Patients and Methods eConsult Selection and Data Abstraction This was a retrospective cohort study conducted on patient eConsults at Mayo Clinic, Rochester, Minnesota, from January 2020 to December 2024. During the study period, an institutional eConsult database contained more than 7000 eConsults referred from general internal medicine (GIM) to all specialty areas. The database was first restricted to eConsults assigned to GIH, leaving approximately 1000 GIH eConsults for further review. Eligibility was initially assessed based on the documented consult indication and subsequently confirmed through manual review of the GIH eConsult. We included eConsults submitted by GIM physicians in which the primary reason for consultation was evaluation or management of MASLD in the setting of abnormal liver function tests (LFTs). Cases were excluded if the primary indication involved other etiologies of abnormal LFTs, including autoimmune hepatitis, primary biliary cholangitis, primary sclerosing cholangitis, viral hepatitis (A, B, C, or E), alcohol-associated liver disease, or drug-induced liver injury. Other exclusions were consults primarily related to genetic or metabolic liver diseases such as hemochromatosis, Wilson disease, α-1 antitrypsin deficiency, acute liver failure, liver transplantation follow-up, or malignancy-associated hepatic dysfunction. Consults for isolated abnormal imaging findings, biliary obstruction, or non-hepatic causes of LFT abnormalities, such as muscle injury, thyroid disease, or cardiac disease, were also excluded. Following this screening process, 125 GIH eConsults remained (see Figure). Flow diagram of eConsult selection and inclusion. A total of 7006 GIM eConsults were identified from January 2020 to December 2024. After filtering to gastroenterology and hepatology consultations (n=1135) and excluding non-MASLD etiologies, 125 MASLD-related eConsults were included in the final analysis. MASLD, metabolic dysfunction-associated steatotic liver disease. The eConsult process at our institution is designed to facilitate asynchronous consultation between GIM physicians and specialists. For each included case, the GIH specialist’s eConsult response was linked to the GIM clinical note that prompted the eConsult. This GIM note, together with laboratory and imaging results available at the time of the eConsult, provided the clinical information used to generate the OpenEvidence consultation. Additional pertinent history, including alcohol, tobacco, and drug use, medications, and past medical and surgical history, was also abstracted from the medical record and included with each case. OpenEvidence-Generated eConsults All OpenEvidence-generated eConsults were produced using the free-to-use LLMs, OpenEvidence. No additional model training was done before the study. Responses were generated during May and June 2025. The specific underlying model version was not displayed by the platform and therefore could not be recorded. Two prespecified prompting strategies were used based on whether the original specialist GI eConsult contained an explicit referral question. Referral questions were present in 116 of 125 specialist GI eConsults. When a referral question was present, the question was inserted verbatim into the following OpenEvidence prompt: “Please provide a focused gastroenterology eConsult for (referral question).” When no explicit referral question was documented, the following standardized prompt was used: “Please provide a focused gastroenterology eConsult for abnormal liver function testing.” Cases were assigned to one of these 2 strategies solely according to the presence or absence of an explicit referral question in the original subspecialty eConsult. Because the original referral order was unavailable, the referral question was extracted from the gastroenterologist-authored eConsult when explicitly documented. Therefore, the absence of a recorded question did not necessarily indicate that the original referral lacked a discrete clinical question. The two-prompt strategy was used to preserve the intent of the original referral when an explicit referral question was available while providing a consistent consultation request when the referral did not contain a discrete question. For each case, OpenEvidence was provided with the deidentified GIM referral note, the alcohol, tobacco, drug, and medication history, and the laboratory and imaging results available at the time of the original consultation. These materials were presented in the same sequence for all cases. Each case was submitted once using the applicable prompt, without follow-up prompting. No user-specified system instructions, model parameters, temperature settings, or other generation controls were added or modified. Platform-internal system instructions and model settings were proprietary and were not visible to the investigators.OpenEvidence specialist eConsults and the output of OpenEvidence-generated eConsults were saved in a REDCap database (REDCap version 15.5.20).23,24 eSQUARE Instrument and Comparison of Specialist and AI eConsults The GIH eConsults and OpenEvidence-generated eConsults were randomized before review. Five physicians (5 GIM physicians) independently evaluated all eConsults for quality using the eSQUARE instrument. The eSQUARE instrument is an 11-item validated tool designed to appraise eConsult quality across 3 domains, General, Management, and Communication, along with a single-item global rating.25 The The eSQUARE instrument evaluates physician reviewers’ perceptions of the quality of the written specialist response across the general, management, and communication domains. Although the instrument includes a management domain, it does not independently establish diagnostic accuracy or assess the quality of underlying clinical reasoning. The instrument reports strong validity evidence, including content validity, internal structure, and evidence of relations with other variables. The original 11-item eSQUARE instrument was administered without modification. (Table 1) The eSQUARE survey contains 3 domains and 1 global quality question. The domains include general (n=4), management (n=4), and communication (n=2). The global question assessed overall quality. A separate study-specific item was added to the REDCap survey asking reviewers to rate the amount of information provided in each eConsult. This supplemental item was analyzed independently and was not included in any eSQUARE domain or total score. Table 1Quality Scoresa,beSQUARE scoreGIH (n=625)OE (n=625)Effect estimate (OE-GI)Estimate (95% CI)PTotal score (maximum 50)26.1 ± 5.832.5 ± 4.26.4 (6.0-6.9)<.001General (maximum 20)11.1 ± 2.413.5 ± 1.82.4 (2.2-2.6)<.001Management (maximum 20)9.5 ± 2.812.4 ± 2.12.9 (2.7-3.1)<.001Communication (maximum 10)5.5 ± 1.26.6 ± 0.91.1 (1.0-1.2)<.001Global rating2 (2-3)3 (3-4)16.1 (12.0-21.5)<.001aAbbreviations: GIH, gastroenterology and hepatology; OE, OpenEvidence.beSquare quality scores comparing GIH and OpenEvidence eConsults. Total, general, management, and communication scores are summarized using mean ± SD. Maximum total score of 50. The global rating was summarized using median (Q1, Q3) and effect estimate corresponds to the OR where an OR >1.0 indicates higher scores for OE relative to GIH. Quality Scoresa,b Abbreviations: GIH, gastroenterology and hepatology; OE, OpenEvidence. eSquare quality scores comparing GIH and OpenEvidence eConsults. Total, general, management, and communication scores are summarized using mean ± SD. Maximum total score of 50. The global rating was summarized using median (Q1, Q3) and effect estimate corresponds to the OR where an OR >1.0 indicates higher scores for OE relative to GIH. All items were rated on a 5-point Likert scale. To establish response process validity, all reviewers jointly evaluated 2 mock eConsults using the eSQUARE instrument and a study-specific item assessing the amount of information. Reviewers discussed item interpretation to ensure consistency. Reviewers were blinded to the consult type (AI vs GIH specialist) during the review process. Both AI and GIH eConsults were reformatted into text documents to reduce stylistic cues. Accuracy of OpenEvidence-Generated eConsults All 5 primary reviewers assessed the accuracy of OpenEvidence-generated eConsults as part of the initial eSQUARE review, with the option to provide free-text comments, which were recorded in REDCap. In addition, one study-author, a board-certified gastroenterologist and hepatologist (DS), served as the expert reviewer and assessed all OpenEvidence-generated eConsults for medical accuracy. The expert review was designed as an exploratory characterization of errors in the OpenEvidence-generated responses rather than as a formally powered or independent safety assessment. Identified concerns were subsequently discussed among the physician coauthors. Because the responses were not independently assessed by multiple hepatologists, the reliability and reproducibility of the error classifications could not be established. For this study, considerable harm was defined as any event that could reasonably result in a delayed diagnosis, grossly inappropriate treatment, hospitalization, or death. Statistical Analyses Total of 125 matched econsult pairs were each evaluated by 5 reviewers. The total, general, management, and communication scores are summarized using mean ± SD and were analyzed with a linear mixed model. The global rating was summarized using the median (Q1, Q3) and was analyzed as an ordinal variable using a generalized linear mixed model with a cumulative logit link function. The amount of information was summarized using n (%) for each category but was analyzed as a binary variable (appropriate amount) using a generalized linear mixed model with a logit link function. For all models, econsult and reviewer were included as random effects, and response type (GIH vs OpenEvidence [OE]) was included as a fixed effect. For the total, general, management, and communication scores, the effect estimate corresponds to the mean difference between response types (OE—GIH). For the global rating, the effect estimate corresponds to the OR, where an OR >1.0 indicates higher OE scores relative to GIH. For the amount of information, the effect estimate corresponds to the OR, where an OR >1.0 indicates higher odds of providing the appropriate amount of information for OE relative to GIH. In addition to the overall analysis, differences between responses (OE vs GIH) were analyzed separately for each reviewer using the paired t test for the total, general, management, and communication scores and the Wilcoxon signed rank test for the global rating. All statistical analyses were performed using SAS software (SAS Institute), with statistical significance defined as P<.05. Because response length differed substantially between GIH and OpenEvidence-generated eConsults, we performed an additional analysis examining the association between word count and eSQUARE total score. Given the minimal overlap in word counts between groups, this association was evaluated separately for GIH-generated and OpenEvidence-generated responses. Linear mixed-effects models were fit with word count modeled using a restricted cubic spline with 4 knots and reviewer included as a random effect. Predicted eSQUARE total scores were plotted across the 2.5th to 97.5th percentile of the observed word count distribution within each group. Ethical Considerations This study was reviewed and approved by the Mayo Clinic institutional review board and deemed to be of minimal risk. All eConsults and referral notes were deidentified following safe harbor procedures.26 All participants had Minnesota Research Authorization, as well. No identifiable patient data was retained during analysis. Results Overall eConsult Quality Across all reviewers, OpenEvidence eConsults received significantly higher eSQUARE-rated perceived quality scores than GIH eConsults. The mean ± SD: 32.5±4.2 vs 26.1±5.8; mean difference, 6.4; 95% CI, 6.0-6.9; P<.001. This was notable in all 3 eSQUARE domains. In the general domain, OpenEvidence-generated eConsults scored higher than GIH eConsults (13.5±1.8 vs 11.1±2.4; mean difference, 2.4; 95% CI, 2.2-2.6; P<.001). Similar findings were noted in the management domain (12.4±2.1 vs 9.5±2.8; mean difference, 2.9; 95% CI, 2.7-3.1; P<.001) and communication domain (6.6±0.9 vs 5.5±1.2; mean difference, 1.1; 95% CI, 1.0-1.2; P<.001) (Table 1). Global Quality Ratings OpenEvidence-generated eConsults received statistically significantly higher scores for global quality than GIH eConsults. The median global rating for OpenEvidence was 3 (IQR, 3-4) compared with 2 (IQR, 2-3) for GIH eConsults. Additionally, OpenEvidence consults had significantly higher odds of receiving a higher global rating than GIH eConsults (OR, 16.1; 95% CI, 12.0-21.5; P <.001). (Table 1). Length of eConsults Perceived length of consultations differed significantly between OpenEvidence-generated eConsults and GIH eConsults. OpenEvidence-generated eConsults were significantly more likely to be rated as appropriate in length compared with GIH eConsults (OR 6.4; 95% CI, 4.8-8.5; P<.001). Table 2 GIH eConsults were more likely to be rated as containing ‘not enough’ (23.4%) or ‘slightly not enough’ information (41.3%). In comparison, OpenEvidence-generated eConsults were more likely to have “slightly too much” (26.1%) or “too much” (1.9%) information. Table 2Perceived Length of Consulta,bAmount of informationGIH (n=625)OE (n=625)Estimate (95% CI)PNot enough146 (23.4%)0 (0.0%)Slightly not enough258 (41.3%)7 (1.1%)Appropriate amount212 (33.9%)443 (70.9%)6.4 (4.8-8.5)<.001Slightly too much7 (1.1%)163 (26.1%)sToo much2 (0.3%)12 (1.9%)aAbbreviations: GIH, gastroenterology and hepatology; OE, OpenEvidence.bPerceived length of eConsult scored from 1=not enough information to 5=too much information. Perceived Length of Consulta,b Abbreviations: GIH, gastroenterology and hepatology; OE, OpenEvidence. Perceived length of eConsult scored from 1=not enough information to 5=too much information. The actual length of documentation also differed between AI and GIH eConsults. GIH eConsults had a mean of 318 words, a median of 295 words, a minimum of 77 words, and a maximum of 1571 words. OpenEvidence-generated eConsults were longer, with a mean of 639 words, a median of 644 words, a minimum of 279 words, and a maximum of 848 words. Response length was also significantly associated with eSQUARE scores within both groups (P≤.001). The median (Q1, Q3) word count was 297 (215-356) words for GIH responses and 668 (619-734) words for OpenEvidence-generated responses, with minimal overlap between the distributions (Supplemental Figure 1, available online at https://www.mcpdigitalhealth.org/). Among GIH responses, eSQUARE total scores increased sharply as word count increased from ∼100 to 300 words, with the association flattening beyond approximately 400 words. OpenEvidence-generated responses had higher word counts, with the 2.5th percentile of the distribution at ∼450 words, and reported a more modest increase in eSQUARE total score across their word count range (Supplemental Figure 2, available online at https://www.mcpdigitalhealth.org/). Reviewer Consistency Individual reviewers reported consistent findings across all domains and global quality ratings, with OpenEvidence-generated eConsults receiving higher scores than GIH eConsults. Although there was heterogeneity in absolute scores among reviewers, the overall pattern of findings was consistent across all reviewers. (Table 3). Table 3Reviewer Specific Resultsa,bScaleGIH (n=125)OE (n=125)PReviewer #1Total score27.6 ± 4.731.7 ± 2.1<.001General11.7 ± 1.912.9 ± 1.1<.001Management10.3 ± 2.212.4 ± 1.2<.001Communication5.6 ± 1.16.3 ± 0.6<.001Global rating3 (2-3)3 (3-3)<.001Reviewer #2Total score21.6 ± 3.933.3 ± 4.6<.001General9.4 ± 1.714.0 ± 2.0<.001Management8.0 ± 1.912.3 ± 2.3<.001Communication4.2 ± 0.87.0 ± 0.9<.001Global rating2 (2-2)4 (3-4)<.001Reviewer #3Total score31.3 ± 6.936.8 ± 4.3<.001General12.8 ± 2.814.8 ± 1.9<.001Management11.9 ± 3.114.6 ± 1.8<.001Communication6.6 ± 1.47.3 ± 1.0<.001Global rating3 (2-4)4 (3-4)<.001Reviewer #4Total score25.4 ± 3.431.0 ± 2.8<.001General11.1 ± 1.413.4 ± 1.5<.001Management8.8 ± 1.911.0 ± 1.4<.001Communication5.6 ± 0.86.6 ± 0.8<.001Global rating2 (2-3)3 (3-4)<.001Reviewer #5Total score24.6 ± 4.729.9 ± 3.1<.001General10.4 ± 2.312.4 ± 1.4<.001Management8.4 ± 2.411.5 ± 1.8<.001Communication5.8 ± 0.76.0 ± 0.4.031Global rating2 (2-3)3 (3-3)<.001aAbbreviations: GIH, gastroenterology and hepatology; OE, OpenEvidence.bTotal, general, management and communication scores were compared using the paired t test. Global Rating was compared using the signed rank test. Reviewer Specific Resultsa,b Abbreviations: GIH, gastroenterology and hepatology; OE, OpenEvidence. Total, general, management and communication scores were compared using the paired t test. Global Rating was compared using the signed rank test. Accuracy of OpenEvidence-Generated eConsults Errors were identified in 13 of 125 OpenEvidence-generated eConsults (10.4%), including 12 responses with minor errors and 1 response with an error classified as potentially major (0.8%). Because this was an exploratory review conducted by a single study-author hepatologist, these estimates should be interpreted with caution. Of the 13 errors, 12 were minor errors. Thematic analysis grouped errors into 4 categories: management recommendation gaps (5 of 12), extraneous information (4 of 12), differential-building issues (3 of 12), and inputting normal results (1 of 12). Extraneous information was defined as statements that were accurate but not applicable to the patient’s clinical presentation. For example, an eConsult might note, “Pioglitazone is another option with proven benefit in NASH…,” for a patient without MASLD, or providing education on vitamin E for biopsy-proven MASLD while simultaneously explaining why it was not recommended. These comments were clinically reasonable but not pertinent to the specific case. Differential-building errors occurred when the eConsult either did not include or failed to expand on an appropriate differential diagnosis. For instance, an eConsult might fail to consider alternative viral etiologies for abnormal LFTs. Management recommendation errors included missing or incorrect guidance that did not pose potential major harm. For example, not addressing an incidentally noted hepatic cyst or omitting vitamin E recommendations for a patient with biopsy-proven MASLD. Imputation of normal results occurred when the eConsult calculated a FIB-4 score assuming a normal ALT value when the actual value had not been specified but had been described as normal in the referring physician’s note. One eConsult was classified as having a major error with potential for considerable harm. This was thematically coded as a gap in management recommendations. The recommended endoscopy for variceal screening in a patient with fibrosis but not known cirrhosis could have led to inappropriate testing and harm related to additional testing. Discussion Comparison of eConsult Quality Scores In this blinded comparative study of 125 paired eConsults for MASLD, OpenEvidence-generated eConsults received higher perceived quality scores than specialist-authored eConsults across all eSQUARE domains and on global ratings. For internal medicine clinicians and health systems, these results suggest that OpenEvidence-assisted drafting may help standardize and scale responses to relatively protocol-driven consultation questions, particularly when specialty access is constrained. At the same time, the detection of clinically relevant errors, including one potentially major error, highlights that OpenEvidence-generated recommendations should be considered decision support rather than autonomous clinical care. Several factors may explain the differences observed. OpenEvidence-generated eConsults were substantially longer and more consistently structured than specialist-authored eConsults, and reviewers more often judged them to contain an appropriate amount of information. In our exploratory analysis, response length was significantly associated with eSQUARE total score within both groups, although the relationship was nonlinear. Among GIH responses, scores increased with increasing length among shorter responses but plateaued at ∼350-400 words, whereas OpenEvidence responses were substantially longer overall and reported only a modest upward trend across their observed range. These findings suggest that response length may contribute to perceived eConsult quality, particularly for shorter specialist responses. However, because few OpenEvidence and GIH responses were of comparable length, we could not determine how much of the difference in eSQUARE scores was from the word count itself. Thus, the higher ratings for OpenEvidence may reflect differences in documentation style, including length, organization, completeness, and explicit reasoning, rather than superior underlying clinical judgment or eConsult length alone. As such, higher eSQUARE Management domain scores should be interpreted as more favorable reviewer assessments of how management recommendations were addressed and communicated within the written response, rather than evidence that the recommendations were more clinically accurate or superior to those of gastroenterologists. If implemented, OpenEvidence-assisted workflows may therefore be most valuable as a mechanism to improve consistency, readability, and completeness of consultation while still requiring clinician review for correctness and appropriateness to the individual patient. Additionally, the comparison evaluated in this study should also not be interpreted as a proposed model of autonomous AI eConsult delivery. In clinical practice, a more plausible workflow would use OpenEvidence to generate an initial draft that is subsequently reviewed, modified, and approved by a human specialist. Such a human-plus-AI model could combine the consistency, structure, and completeness observed in OpenEvidence-generated responses with specialist judgment, contextual knowledge, and accountability. However, the present study did not evaluate this combined workflow and therefore cannot determine whether OpenEvidence assistance improves final response quality, reduces specialist time or cognitive burden, or decreases or introduces clinical errors. Implications for Clinical Practice First, for common, relatively protocol-driven consultation questions, such as MASLD risk stratification and initial management, OpenEvidence may be most appropriately evaluated as a drafting and decision support tool rather than as an autonomous eConsult provider. OpenEvidence-generated drafts could provide a structured starting point for specialists, reduce variability in consultation content, and make the rationale for recommendations more explicit for referring clinicians. Because occasional errors may not be apparent to nonspecialists, the specialist should remain responsible for verifying the accuracy, relevance, and patient-specific appropriateness of the final response. Second, OpenEvidence-generated eConsults consistently followed a structured format, including headers, bullet points, and numbered problem lists. In contrast, GIH eConsults more commonly consisted of longer narrative text, often without separation between clinical assessment and management recommendations. Prior studies on clinical documentation suggest that structured formats improve readability and may be perceived as higher quality.27,28 Third, experienced specialists may be subject to the ‘curse of knowledge’ bias, a common cognitive bias where experts, because of their familiarity with a topic, have difficulty recognizing information gaps and identifying the needs of less specialized clinicians.29 Consequently, specialists may omit intermediate reasoning steps that are thought to be obvious within the specialty but are less obvious to clinicians outside the specialty or in primary care. In comparison, OpenEvidence-generated eConsults often made reasoning more explicit, providing clear details regarding differential diagnoses, risk assessment, and management decisions. This may enhance clarity and perceived usefulness, particularly when evaluating eConsults using a quality assessment instrument, such as eSQUARE. Fourth, GIH eConsults were generated from 2020 to 2024, during a period of evolving terminology and management for MASLD. OpenEvidence-generated eConsults were produced at a single time point and may have reflected more current guideline language. Regardless, evolving standards of care remain a potential source of bias, and future work should incorporate temporally matched comparisons to further mitigate this concern. Fifth, the novelty of this study is incremental. Previous research has already reported that LLM-generated responses may have positive reviewer ratings for communication and clinical tasks. Our findings add to that growing body of literature by evaluating a larger set of paired, real-world physician-to-specialist eConsults using an eConsult-specific quality instrument. The use of real-world referrals and cases parallels real clinical environments where specialists may review eConsults with incomplete information. Similarly, use of eSQUARE provides a structured assessment focusing on the quality of correspondence rather than a perceived satisfaction or global preference score. These refinements do not establish superior clinical reasoning or broad generalizability across specialties. However, they do provide evidence regarding the perceived quality and potential safety limitations of OpenEvidence-generated eConsult for MASLD. Consistency Among Reviewers Our findings report consistent patterns of consult quality across all reviewers. Although there was heterogeneity in absolute scores, each reviewer identified statistically significant differences between OpenEvidence-generated and GIH eConsults across all domains and global quality ratings (Table 1). This suggests that reviewers differed in their overall scoring stringency while the direction of the comparison remained consistent across reviewers. Notably, absolute scores in this study were lower than those reported in the original eSQUARE validation study.25 Several factors may explain this difference. First, reviewer calibration may have influenced scoring behavior. All reviewers in our study jointly evaluated 2 mock eConsults and were instructed to avoid ceiling effects by applying the full Likert scale thoughtfully rather than defaulting to higher scores. Second, the eSQUARE Likert scale ranges from 1 (not at all) to 5 (exemplary), and reviewers may have opted to avoid labeling OpenEvidence-generated or GIH eConsults as truly exemplary, resulting in lower overall scores. The original eSQUARE validation study reported that the instrument could distinguish among eConsults categorized as low, moderate, or high quality. However, it did not establish validated score thresholds or a minimally important difference for these values. Therefore, although the observed differences were consistent across reviewers, the practical or clinical importance of the magnitude of those differences cannot be determined from the available evidence. OpenEvidence-generated eConsult Errors The exploratory expert review identified errors in 13 of 125 OpenEvidence-generated eConsults, including one error classified as potentially major. These findings report that clinically relevant errors may occur even when OpenEvidence-generated responses receive favorable ratings for perceived eConsult quality. However, the observed frequency should not be interpreted as an estimate for the safety of OpenEvidence. The review was performed by a single study-author hepatologist, was not independently duplicated, and did not include a formal adjudication process. In addition, we evaluated only OpenEvidence-generated eConsult accuracy from a safety perspective because it represents a novel care-delivery model. In contrast, GI eConsults are an established model with substantial evidence supporting their accuracy and clinical utility.30 Therefore, the study cannot determine whether errors occurred more or less frequently in OpenEvidence-generated responses than in gastroenterologist-authored responses. The error assessment should instead be interpreted as exploratory and as a potential future area for study. Limitations This study has several limitations. First, as a retrospective single-center study at a tertiary referral center, findings may not generalize to community practice settings or to health systems with different eConsult workflows. Second, we focused on MASLD, a relatively protocol-driven consultation question with well-described noninvasive risk stratification and management pathways. Performance may differ for less protocol-driven consult questions. Third, this study evaluated only OpenEvidence, a proprietary platform, using standardized prompts at a single point in time. Performance may differ across platforms, model versions, specialties, clinical conditions, input formats, and prompting strategies. These findings should be interpreted as specific to OpenEvidence and should not be generalized to all LLM-generated eConsults. Fourth, the additional item assessing the amount of information has not been validated specifically for OpenEvidence-generated eConsults. Fifth, eSQUARE assesses physician reviewers’ perceptions of the quality of specialist correspondence. Higher total or domain scores do not establish greater diagnostic accuracy, superior management decisions, or superior underlying clinical reasoning. The study was also not designed to assess downstream utilization, referral avoidance, time to resolution, or patient-level safety outcomes. Sixth, although OpenEvidence generates cited recommendations, we removed citations before blinded review to reduce source-identifying cues; therefore, we did not assess citation validity or the risk of hallucinated sources. Seventh, human eConsults were written from 2020 to 2024 under real-world clinical time constraints. Furthermore, MASLD terminology and guidance evolved during the study period (eg, transition from nonalcoholic fatty liver disease to MASLD), and specialist eConsults authored between 2020 and 2024 may reflect temporal heterogeneity not present in OpenEvidence-generated responses created at a single time point. It is possible that these nonequivalent conditions may have biased the comparison in favor of OpenEvidence. Finally, the safety assessment was exploratory and was based on 125 OpenEvidence-generated responses reviewed by a single study-author hepatologist. Although identified concerns were discussed among the physician coauthors, the review was not independently duplicated and did not include a formal adjudication process. The error classifications may therefore be subject to reviewer bias, and their interrater reliability could not be assessed. In addition, the study was not powered to estimate uncommon but potentially serious errors. As such, the reported error frequencies should not be interpreted as generalizable estimates of platform safety. Conclusion OpenEvidence-generated eConsults received higher ratings of perceived quality and informational adequacy than gastroenterologist-authored responses for MASLD, although clinically relevant errors were identified. OpenEvidence-generated eConsults may help clinicians in resource-constrained environments where access to GIH in-person consultations is limited. However, these findings do not support the autonomous delivery of OpenEvidence-generated eConsults. Rather, they provide a rationale using OpenEvidence as a drafting or decision support tool with specialist oversight. Future studies should compare specialist-only eConsults with OpenEvidence-assisted specialist eConsults and assess final response quality, safety, editing burden, time savings, and clinician acceptance. Potential Competing Interests Dr Hurt is a consultant for Nestlé Nutrition for research activities unrelated to the content of this manuscript. No author has any financial, employment, equity, advisory, consulting, research-funding, or other professional relationship with OpenEvidence. OpenEvidence personnel provided limited technical assistance with platform access and configuration but had no role in the study design, case selection, data collection, response evaluation, statistical analysis, interpretation of the results, manuscript preparation, or decision to submit the manuscript for publication. All other authors declare no financial or nonfinancial relationships or activities that could appear to have influenced the submitted work. Ethics Statement This study was reviewed and approved by the Mayo Clinic Institutional Review Board and deemed to be of minimal risk. Acknowledgments The authors thank Eric Lehman and Zack Ziegler of OpenEvidence for limited technical assistance with platform access and configuration. OpenEvidence and its personnel had no role in the study design, case selection, data collection, response evaluation, statistical analysis, interpretation of the results, manuscript preparation, or decision to submit the manuscript for publication. The authors also thank Shawn Fokken of Mayo Clinic for assistance with REDCap database setup and data-management support. Footnotes Supplemental material can be found online at https://www.mcpdigitalhealth.org/. Supplemental material attached to journal articles has not been edited, and the authors take responsibility for the accuracy of all data. Supplemental Online Material Supplemental Figures References Associated Data Supplementary Materials Supplemental Figures