Evaluating search-enabled large language model interfaces for mpox public health consultation: a guideline-based comparative study Authors: Qiqi Zheng, Ru Chen, Mingming Cai, Yanyan Chen, Xiuli Lin, Tingting Wang Published: 2026 ABSTRACT BackgroundSearch-enabled large language model interfaces are increasingly used by the public for health information, but their performance in mpox-related public health consultation remains unclear. This study evaluated their safety, accuracy, empathy, reliability/information quality, and readability.MethodsWe conducted a single-query comparative cross-sectional evaluation using 52 predefined mpox-related public consultation questions. Each question was submitted once to each of six search-enabled LLM interfaces, yielding 312 first responses. Responses were assessed against a guideline-based reference framework. Safety was coded as a binary outcome, while accuracy and empathy were rated on 5-point scales. Reliability/information quality was evaluated using DISCERN, EQIP, JAMA benchmark criteria, and GQS. Readability was assessed using six established readability indices. Five trained raters independently evaluated the human-scored outcomes.ResultsUnsafe responses were relatively infrequent but occurred in all six interfaces, with safe-response rates ranging from 86.5 to 92.3%. No pairwise difference in Safety remained statistically significant after Benjamini–Hochberg correction. Overall differences across interfaces were statistically significant for Accuracy, Empathy, all four reliability/information quality measures, and all six readability indices. Benjamini–Hochberg-adjusted post hoc analyses identified outcome-specific pairwise differences, although the pairwise patterns varied across measures.ConclusionThe evaluated search-enabled LLM interfaces showed heterogeneous performance across safety, accuracy, empathy, reliability/information quality, and readability. Although unsafe responses were relatively uncommon, potentially harmful outputs occurred in every interface. These findings support the need for guideline-based evaluation, source transparency, readability optimization, and robust safety safeguards when such interfaces are evaluated or considered for mpox-related public health consultation. The results represent a time- and configuration-specific interface-level snapshot; they should not be attributed to the underlying base models in isolation or interpreted as establishing reproducible performance or a stable hierarchy across sessions, versions, or settings. Introduction Mpox is an Orthopoxvirus infection caused by the monkeypox virus and is characterized by systemic symptoms and skin or mucosal lesions that may last for several weeks (1). The 2022 multinational outbreak substantially changed its public-health profile, with widespread transmission reported in countries without previous sustained circulation and many cases occurring through close physical or sexual contact networks (2, 3). Subsequent outbreaks involving clade I viruses in Central and Eastern Africa, together with international spread and imported cases, have kept mpox on the global public-health agenda (4, 5). The World Health Organization’s recommendation to use the term “mpox” also highlights the importance of non-stigmatizing communication in disease naming, risk communication, and public trust (6). Public consultation about mpox is particularly complex because appropriate advice must integrate symptom recognition, differential diagnosis, transmission risk, post-exposure management, vaccination, treatment, infection control, and stigma-sensitive communication. Mpox can mimic other rash illnesses, including varicella, herpes zoster, herpes simplex infection, syphilis, and nonspecific dermatitis, making visual self-diagnosis unreliable. Transmission advice also requires nuance: mpox is mainly transmitted through close contact with lesions, body fluids, contaminated materials, and prolonged close exposure to respiratory secretions, rather than through brief casual contact in the manner of highly airborne respiratory infections (1, 7). In addition, recommendations regarding post-exposure vaccination, antiviral use, isolation, care seeking, and local risk assessment may depend on exposure timing, exposure type, disease severity, risk group, vaccine availability, jurisdiction, and current surveillance data (8–11). Therefore, public-facing mpox advice must be accurate, safe, actionable, up to date, understandable, and non-stigmatizing. Large language model (LLM) interfaces are increasingly used by the public as accessible digital health information tools. These systems can generate fluent answers to medical questions, translate technical content into lay language, and provide seemingly personalized guidance at low cost and high speed. Previous studies have shown that LLMs can encode substantial clinical knowledge and may produce patient-facing responses that are rated highly for quality and empathy in some settings (12, 13), and their use in patient education has been explored across medical specialties (14). However, systematic reviews and meta-analyses have raised concerns about variable accuracy, hallucinated or unsupported claims, inconsistent reporting methods, and potentially harmful health advice (15, 16). These concerns are especially relevant for infectious diseases, where inaccurate or over-reassuring advice may delay testing, isolation, vaccination, clinical assessment, or public-health action, and may also amplify stigma or misinformation. From a digital public health perspective, public-facing LLM interfaces can be understood as interactive health information tools that combine natural-language generation, web retrieval, interface-level source presentation, and safety mechanisms. Their evaluation should therefore extend beyond factual correctness alone. The CHART statement emphasizes transparent reporting of model identity, query strategy, reference standards, raw outputs, performance evaluation, and potentially harmful, biased, or misleading responses (17). For mpox-related public consultation, evaluation should extend beyond factual correctness because responses may contain diverse safety-relevant problems, such as over-reassurance after exposure, unsupported jurisdiction-specific risk estimates, overstated vaccine or antiviral effects, minimization of contaminated-material risks, or stigmatizing communication. These examples were considered potential manifestations of unsafe content rather than prespecified quantitative safety subdomains. In the present study, Safety was evaluated as a binary primary outcome, with category-specific patterns examined descriptively in a post hoc analysis of responses classified as unsafe. Similarly, technically accurate information may be of limited public-health value if it is difficult for lay readers to understand or lacks empathy when addressing anxiety, privacy, sexual contact, pregnancy, immunocompromise, or pediatric concerns. Readability and clear communication are therefore central to the usefulness of digital public health information (18, 19). Despite the rapid expansion of LLM applications in health information delivery, evidence remains limited regarding their performance in mpox-specific public health consultation. In particular, few studies have compared multiple widely accessible search-enabled LLM interfaces using a guideline-grounded question set that reflects real public concerns while simultaneously evaluating safety, accuracy, empathy, reliability/information quality, and readability. To address this gap, the present study assessed six publicly accessible search-enabled LLM interfaces—ChatGPT, Gemini, Copilot, DeepSeek, Doubao, and Qwen—using a locked set of 52 public-facing mpox consultation questions. Responses were evaluated against a predefined guideline-based reference framework. This study aimed to characterize the response-level performance and limitations of six publicly accessible search-enabled LLM interfaces in mpox-related public health consultation. Materials and methods Study design and reporting framework This comparative cross-sectional study evaluated six publicly accessible search-enabled LLM interfaces using standardized public-facing mpox consultation questions. The target use context was members of the general public seeking mpox-related information through consumer-facing chatbot interfaces. Performance was assessed across Safety, Accuracy, Empathy, reliability/information quality, and readability. Reporting followed the CHART statement (17). The overall study workflow is shown in Figure 1. Study workflow. Development of the mpox public consultation question set The question set was developed through purposive multi-source identification of public search-intent signals, online consultation content, patient-education resources, and clinical and public-health evidence. One infectious-disease physician and one infectious-disease nurse collected and organized candidate questions from Google Trends, Baidu Index and Baidu Search, public forums and Q&A platforms, patient-education resources, guidelines, consensus documents, and reviews. Detailed platforms, URLs, search terms, settings, access dates, and source-specific counts are provided in Supplementary Material 1. The initial search yielded 168 pre-deduplication signals. After deduplication, merging of overlapping items, and rewriting into natural public-consultation language, 120 candidate questions remained. The same two researchers then independently screened the questions for relevance and scoreability, excluding 31 items, followed by a second-stage assessment of priority and balance across 12 predefined domains, excluding a further 37 items. Disagreements were resolved through discussion, with unresolved items adjudicated by a senior infectious-disease physician. The final locked set comprised 52 questions. Inter-screener raw agreement was 92.5%, with a Cohen’s κ of 0.798 at the first stage, and 86.5%, with a Cohen’s κ of 0.720 at the second stage. Complete screening criteria, agreement results, item-level inclusion and exclusion decisions, reasons for exclusion, and domain allocation are provided in Supplementary Material 1. The final question set was developed through purposive, coverage-oriented selection and was intended to capture common and safety-sensitive mpox consultation scenarios rather than constitute a probabilistically representative sample of all public mpox queries. Guideline-based reference framework An investigator-developed guideline-based reference framework was constructed before formal response evaluation by one infectious-disease physician and one infectious-disease specialist nurse, neither of whom participated in formal response scoring. The framework was developed specifically for this study and was not an independently validated gold standard. For each of the 52 questions, it specified a suggested public-facing answer, the corresponding guideline content or rationale, and the classification of supporting evidence. Formal guidelines, national guidance, WHO, CDC, and ACIP recommendations, and expert consensus statements were prioritized, whereas reviews and public-education materials were used only as supplementary evidence or to support public-facing wording. During manuscript revision, two external experts from an institution independent of the author team—one infectious-disease specialist and one infection prevention and control specialist—independently reviewed all 52 entries for accuracy, completeness, guideline concordance, source appropriateness, safety, uncertainty, and temporal or jurisdictional applicability. Both experts accepted all entries without revision. This external review provided an additional check of content and applicability but did not constitute formal validation of the framework. Further details are provided in Supplementary Material 2. LLM interface selection and query procedure Six publicly accessible search-enabled LLM interfaces were evaluated: ChatGPT, Gemini, Microsoft Copilot, DeepSeek, Doubao, and Qwen. The providers, interface-displayed model or mode labels, official access routes, account and subscription status, search or retrieval settings, and archived documentation are summarized in Table 1. Because consumer-facing interfaces may use dynamic routing or undisclosed backend updates, the displayed labels should not be interpreted as independently verified fixed backend model versions. Table 1Interface identifiers, access conditions, and query documentation.InterfaceProviderInterface-displayed model/versionOfficial web access and accountSelected reasoning/retrieval settingBackend disclosureChatGPTOpenAIGPT-5.5 Pro, as displayedhttps://chatgpt.com/; logged-in paid consumer accountGPT-5.5 Thinking manually selected; Web Search enabled according to the query protocolProprietary hosted interface; fixed backend weights not independently verifiableGeminiGoogle/Google DeepMindGemini 3.0 Deep Think, as displayedhttps://gemini.google.com/app; logged-in Google AI Ultra accountDeep Think manually selected; search-enabled setting recorded in the query logProprietary hosted interface; dynamic routing may occurMicrosoft CopilotMicrosoftMicrosoft Copilot; underlying model not disclosedhttps://copilot.microsoft.com/; logged-in free consumer accountSearch conversation mode manually selected; web retrieval with citationsProprietary hosted interface; underlying model version not disclosedDeepSeekDeepSeekDeepSeek-V4-Flash, as displayedhttps://chat.deepseek.com/; logged-in free consumer accountDeep Think and Web Search manually enabledHosted interface; exact backend routing not independently verifiableDoubaoByteDanceDoubao-Seed-2.0, as displayedhttps://www.doubao.com/chat/; logged-in free consumer accountExpert/deep-thinking mode and Web Search manually enabledProprietary hosted interface; exact routed backend should not be inferred unless displayedQwenAlibaba CloudQwen3.5, as displayedhttps://tongyi.aliyun.com/qianwen/; logged-in free consumer accountThinking mode manually selected; Web Search/tool functions enabledHosted consumer interface; exact routed backend not independently verifiableAll interfaces were queried between May 1 and May 26, 2026, in Wenzhou, Zhejiang, China. Model/mode labels and selected reasoning or retrieval settings were recorded from the consumer interfaces in the contemporaneous query log; because backend routing was not independently verifiable, these labels should be interpreted as interface-displayed identifiers. Raw outputs and query logs were retained by the study team; standardized structured response records are provided in Supplementary Material 3, and contemporaneous interface screenshots were not retained. Interface identifiers, access conditions, and query documentation. All interfaces were queried between May 1 and May 26, 2026, in Wenzhou, Zhejiang, China. Model/mode labels and selected reasoning or retrieval settings were recorded from the consumer interfaces in the contemporaneous query log; because backend routing was not independently verifiable, these labels should be interpreted as interface-displayed identifiers. Raw outputs and query logs were retained by the study team; standardized structured response records are provided in Supplementary Material 3, and contemporaneous interface screenshots were not retained. All queries were conducted between May 1 and May 26, 2026, in Wenzhou, Zhejiang, China. Each of the 52 questions was submitted once to each interface, using identical wording, in a new independent chat session. Prior session context was cleared before each query. The prespecified search or retrieval functions listed in Table 1 were enabled, and no additional role prompt, prompt engineering, follow-up question, repeated-query run, or manual correction was applied. The initial response was saved immediately after generation. This procedure yielded 312 interface question responses. Accordingly, the study assessed first response performance during the predefined query period rather than repeated-query reproducibility or temporal stability. Detailed query documentation and the complete structured response record are provided in Supplementary Material 3. Benchmark size and domain coverage The benchmark size was determined a priori according to domain coverage and evaluation feasibility rather than by a conventional clinical power calculation. The final set comprised 52 questions across 12 predefined domains, with four to five questions per domain to cover both common information needs and safety-sensitive scenarios. Because every question was evaluated across all six interfaces, statistical comparisons were paired at the question level. Aggregation of rater scores For Safety, each rater independently classified every response as safe or unsafe. The final analytic status was determined using the prespecified majority-vote rule, with a response classified as unsafe when at least three of five raters assigned an unsafe rating. This consensus-based threshold was used to provide a consistent response-level outcome and reduce the influence of an isolated rating. Responses receiving fewer than three unsafe ratings were classified as not meeting the primary unsafe criterion and should not be interpreted as having been unanimously judged safe. Complete de-identified rater-level Safety ratings are provided in Supplementary Material 6. For Accuracy, Empathy, DISCERN, EQIP, JAMA, and GQS, the arithmetic mean of the five independent ratings was calculated for each question–interface pair to provide one analytic score and reduce the influence of any individual rater. For outcomes originating from ordinal scales, this mean was used only for rater aggregation; subsequent comparisons used rank-based methods and did not assume normality or interval-level measurement. Use of raw and cleaned interface outputs Raw interface outputs were generated and archived by one infectious-disease nurse. After completion of data collection, a second nurse researcher who was not involved in question-set development, guideline-framework development, response evaluation, or statistical analysis independently prepared the standardized rating dataset according to predefined cleaning rules. Actual interface names were replaced with coded labels (Model 1–Model 6), and identifiable platform-specific and non-substantive formatting artifacts were removed, while substantive response content and source-attribution information required for scoring were retained. The cleaned records were checked against the archived raw outputs for completeness and accuracy, and the interface-code key was stored separately from the rater-facing materials. A separate text-only copy was prepared for readability analysis. URLs, DOI or PMID strings, reference lists, isolated citation markers, emojis, broken formatting symbols, and image-only content were removed where identifiable to avoid artificially increasing readability difficulty. All human-scored outcomes were based on the interface-identity-masked rating copy rather than the readability-specific copy. Interpretation of interface-level findings The study evaluated publicly accessible, search-enabled chatbot interfaces as configured during the query period, rather than the intrinsic performance of underlying base models. Accordingly, all findings should be interpreted as interface-level results under the recorded access routes, retrieval settings, geographic location, and query dates. Response evaluation process Five raters independently evaluated the interface responses, including two infectious-disease physicians and three infectious-disease nurses. They were not involved in question set development, guideline framework development, interface querying, or preparation of the rating materials. Before formal evaluation, all raters received standardized training and completed a pilot scoring exercise to harmonize interpretation of the scoring criteria. Actual interface names were replaced with coded labels (Model 1–Model 6), and the code key was not provided to the raters. Responses were presented in question-level blocks, but the six coded responses within each block were not formally randomized. Although explicit interface identifiers and identifiable platform-specific cues were removed where possible, complete assessor blinding could not be guaranteed because the originating interface might still be inferred from linguistic style, response structure, or source-presentation patterns. Each rater independently scored all responses using the predefined guideline-based reference framework and scoring rubric. Further details on rater characteristics and agreement results are provided in Supplementary Material 4. Evaluation metrics and scoring All interface responses were evaluated against the predefined investigator-developed guideline-based reference framework. The evaluation covered five domains: safety, accuracy, empathy, reliability/information quality, and readability. The evaluation framework followed the CHART recommendation that studies of chatbot health advice should report reference standards, evaluator processes, and potentially harmful, biased, or misleading responses (17). Safety was the prespecified binary primary outcome; the majority-vote aggregation rule is described in Section 2.6. All responses meeting this criterion were included in a post hoc exploratory reassessment; none were sampled or excluded. As part of this post hoc analysis, recurrent concerns across the complete unsafe-response set were reviewed and consolidated into eight guideline-aligned safety themes. These themes were developed for post hoc descriptive characterization of the unsafe-response set and were not prespecified secondary outcomes or used for inferential interface comparisons. After the thematic framework had been finalized, the same five raters independently reviewed the responses in random order with interface identifiers removed and assigned a severity score (0 = no meaningful potential harm, 1 = minor, 2 = moderate, and 3 = major), one primary safety theme, and plausible clinical or public-health consequences. Final severity was determined by the median of the five ratings, whereas the primary theme was determined by agreement of at least three raters, with unresolved theme classifications adjudicated by a senior infectious-disease physician. Accuracy was rated on a 5-point ordinal scale according to concordance with the guideline-based reference framework: 1 = completely inaccurate or misleading, 2 = mostly inaccurate, 3 = partially accurate, 4 = mostly accurate with minor omissions, and 5 = fully accurate and guideline-concordant. A 5-point expert rating format was used because previous chatbot health-advice studies have used ordinal expert ratings to assess response quality and clinical usefulness (12). Empathy was rated on a 5-point scale: 1 = not empathetic, 2 = minimally empathetic, 3 = moderately empathetic, 4 = empathetic, and 5 = highly empathetic. Ratings considered whether responses were respectful, supportive, privacy-preserving, non-stigmatizing, and appropriately reassuring, informed by previous chatbot evaluation studies and clinical empathy assessment concepts (12, 20). Reliability/information quality was evaluated using four tools. DISCERN assessed the reliability and quality of written consumer health information (21). EQIP assessed the quality of written patient information (22). JAMA benchmark criteria assessed transparency using authorship, attribution, disclosure, and currency (23). DISCERN, EQIP, and the JAMA benchmarks were originally developed for written consumer health-information resources. Because each first response was evaluated as a standalone consumer-facing text, these instruments were considered appropriate for assessing the quality and transparency of the displayed information. No item wording, scoring criteria, weighting rules, or total-score calculations were modified. All items were scored according to information observable in the archived response, with absent information scored as unmet under the original scoring frameworks. Similar multidimensional frameworks combining established health-information quality instruments with readability indices have been applied in recent evaluations of chatbot-generated health information across clinical and public-health contexts (24, 25). GQS provided an overall 1–5 rating of information flow, completeness, and usefulness (26). Readability was assessed using six established indices: Flesch Reading Ease Score, Flesch–Kincaid Grade Level, SMOG, Automated Readability Index, Coleman–Liau Index, and Gunning Fog Index (27–32). Higher Flesch Reading Ease scores indicate easier readability, whereas lower scores for the other five indices indicate easier readability. Readability indices were calculated from the cleaned text-only outputs using the quanteda package and related text-statistics functions (33). Statistical analysis All statistical analyses were performed using R software (version 4.6.1). Safety was summarized as numbers and percentages; Accuracy, Empathy, DISCERN, EQIP, JAMA, and GQS as medians with interquartile ranges; and readability indices as means with standard deviations. Because several human-scored outcomes originated from ordinal scales, rank-based methods were used without assuming normality or interval-level measurement. Because the same 52 questions were submitted to all six interfaces, comparisons were conducted as paired analyses at the question level. Overall differences in non-binary outcomes were assessed using the Friedman test, followed by paired Wilcoxon signed-rank tests. Safety was analyzed using Cochran’s Q test, followed by paired McNemar tests. Pairwise p values were adjusted within each outcome using the Benjamini–Hochberg method (34). Kendall’s W and paired rank-biserial correlations were reported as effect sizes for overall and pairwise non-binary comparisons, respectively (35). For Safety, Q/[n × (k − 1)] was reported as the standardized overall effect measure, whereas paired safe-rate differences and matched odds ratios were reported for pairwise comparisons. Effect estimates were accompanied by 95% confidence intervals, with bootstrap intervals based on 3,000 question-level resamples where applicable (36). Inter-rater agreement was assessed using Fleiss’ κ for Safety (37) and two-way random-effects, absolute-agreement, single-measure intraclass correlation coefficients [ICC(2,1)] for Accuracy, Empathy, DISCERN, EQIP, JAMA, and GQS (38). The post hoc severity and thematic findings were summarized descriptively; complete de-identified rater-level Safety ratings are provided in Supplementary Material 6. Ordinal Krippendorff’s α was used to assess agreement on severity ratings (39). No inferential interface comparisons or rankings were performed for the exploratory safety analysis. All tests were two-sided, and adjusted p values <0.05 were considered statistically significant. Results Inter-rater agreement Five raters independently evaluated all human-scored outcomes. Inter-rater agreement was high across all evaluated domains. For Safety, Fleiss’ κ was 0.869 (95% CI, 0.800–0.923; p < 0.001). For the remaining human-scored metrics, ICC (2,1) values ranged from 0.828 to 0.883, including Accuracy, Empathy, DISCERN, EQIP, JAMA, and GQS. All agreement estimates were statistically significant (all p < 0.001), supporting the consistency of subsequent interface-level comparisons. Detailed inter-rater agreement results are provided in Supplementary Material 4. Safety, accuracy, and empathy Unsafe responses were identified in all six interfaces but were relatively infrequent, with safe-response rates ranging from 86.5 to 92.3%. The overall difference in Safety was not statistically significant (Cochran’s Q = 2.470, df = 5, p = 0.781), and no pairwise Safety comparison remained significant after Benjamini–Hochberg correction. Accordingly, the observed numerical variation should be interpreted descriptively and does not establish differential safety performance across interfaces. Interface-specific safe-response rates are presented in Table 2. Table 2Safety, accuracy, and empathy scores across interfaces.MetricChatGPTGeminiCopilotDeepSeekDoubaoQwenOverall PEffect sizeSafe responses, n (%)48 (92.3)a47 (90.4)a46 (88.5)a47 (90.4)a48 (92.3)a45 (86.5)a0.7810.010Accuracy, median [IQR]4.00 [4.00, 4.00]a4.00 [3.80, 4.00]a3.00 [3.00, 3.00]b3.00 [3.00, 3.00]b3.00 [3.00, 3.00]b3.00 [2.80, 3.00]c<0.0010.694Empathy, median [IQR]4.00 [4.00, 4.00]a4.00 [4.00, 4.00]a3.00 [3.00, 3.00]b3.00 [3.00, 3.00]b3.00 [3.00, 3.00]b2.80 [2.00, 3.00]c<0.0010.843Safety is presented as n (%), whereas Accuracy and Empathy are presented as median [IQR]. Overall comparisons used Cochran’s Q test for Safety and Friedman tests for Accuracy and Empathy. Overall effect sizes are Q/[n × (k − 1)] for Safety and Kendall’s W for Accuracy and Empathy. Values within the same row sharing at least one superscript letter did not differ significantly; values with no letter in common differed significantly in Benjamini–Hochberg-adjusted paired comparisons. Letters are statistical grouping labels only and should not be interpreted as interface rankings. Complete numerical pairwise estimates and 95% confidence intervals are provided in Supplementary Material 4. Safety, accuracy, and empathy scores across interfaces. Safety is presented as n (%), whereas Accuracy and Empathy are presented as median [IQR]. Overall comparisons used Cochran’s Q test for Safety and Friedman tests for Accuracy and Empathy. Overall effect sizes are Q/[n × (k − 1)] for Safety and Kendall’s W for Accuracy and Empathy. Values within the same row sharing at least one superscript letter did not differ significantly; values with no letter in common differed significantly in Benjamini–Hochberg-adjusted paired comparisons. Letters are statistical grouping labels only and should not be interpreted as interface rankings. Complete numerical pairwise estimates and 95% confidence intervals are provided in Supplementary Material 4. All 31 responses meeting the prespecified majority-vote unsafe criterion underwent a post hoc exploratory reassessment. None were classified as having no meaningful potential harm; 5 (16.1%) were rated as minor, 24 (77.4%) as moderate, and 2 (6.5%) as major potential harm. The most frequent primary themes were transmission, environmental-risk, or infection-control misinformation (10/31, 32.3%); unverified jurisdiction-specific or time-sensitive information (6/31, 19.4%); and vaccine-related misinformation or overstatement (5/31, 16.1%). The complete descriptive distribution across all eight post hoc safety themes is provided in Supplementary Material 5. These theme-specific findings were not treated as prespecified secondary outcomes and were not used for inferential interface comparisons or rankings. Illustrative examples included one major-harm response stating that a mother with suspected mpox could continue direct breastfeeding without adequately clarifying the required conditions and precautions, potentially allowing close skin-to-skin exposure before clinical or public-health assessment. One moderate-harm response provided unverified location-specific outbreak estimates without first confirming the user’s country, which could misdirect decisions regarding travel, testing, vaccination, or care seeking. Complete item-level classifications, including unsafe-vote counts, severity distributions, primary themes, plausible consequences, and safer-response directions, are provided in Supplementary Material 5. Accuracy and Empathy differed significantly across interfaces. Benjamini–Hochberg-adjusted pairwise analyses placed ChatGPT and Gemini in one statistical group, Copilot, DeepSeek, and Doubao in a second group, and Qwen in a separate lower-scoring group for both outcomes. These groupings describe the present response set and should not be interpreted as stable interface rankings. The distributions of Safety, Accuracy, and Empathy are shown in Figure 2, and the corresponding descriptive results, overall comparisons, and adjusted pairwise groupings are summarized in Table 2. Complete numerical pairwise results, including effect estimates, 95% confidence intervals, and adjusted p values, are provided in Supplementary Material 4. Distributions of safety, accuracy, and empathy across interfaces. Reliability/information quality Reliability and information-quality measures differed significantly across interfaces, although the pairwise patterns varied by measure. DISCERN and EQIP yielded distinct Benjamini–Hochberg-adjusted groupings across the six interfaces, whereas JAMA and GQS showed partial overlap among interface groups. These findings represent measure-specific differences within the present response set rather than a stable overall hierarchy. Figure 3 presents the score distributions, and Table 3 summarizes the descriptive results, overall effect sizes, and adjusted pairwise groupings. Complete numerical pairwise estimates are provided in Supplementary Material 4. Distributions of reliability/information quality scores across interfaces. Table 3Reliability/information quality scores across interfaces.MetricChatGPTGeminiCopilotDeepSeekDoubaoQwenOverall PKendall’s WKendall’s W (95% CI)DISCERN60.10 [59.15, 61.60]a54.00 [52.60, 55.25]b47.60 [46.55, 48.80]c45.20 [43.55, 46.40]d41.60 [40.60, 42.60]e38.20 [37.00, 39.20]f<0.0010.9540.927–0.978EQIP65.50 [64.00, 67.00]a60.00 [58.00, 61.00]b49.00 [48.00, 51.00]c45.00 [42.00, 46.00]d42.00 [40.00, 43.00]e36.00 [34.00, 37.00]f<0.0010.9310.907–0.956JAMA2.00 [2.00, 2.00]a2.00 [2.00, 2.00]a2.00 [1.00, 2.00]b1.00 [1.00, 1.00]c1.00 [1.00, 1.00]c1.00 [1.00, 1.00]c<0.0010.7390.698–0.789GQS4.00 [4.00, 4.00]a4.00 [3.80, 4.00]b3.00 [3.00, 3.00]c3.00 [3.00, 3.00]c3.00 [3.00, 3.00]c3.00 [2.80, 3.00]d<0.0010.7500.677–0.826Values are median [IQR]. Within each row, values sharing at least one superscript letter did not differ significantly; values with no superscript letter in common differed significantly in paired Wilcoxon signed-rank tests after Benjamini–Hochberg adjustment. Overall p-values were obtained using Friedman tests, and Kendall’s W is reported with its bootstrap 95% CI. Letters are grouping labels only and should not be interpreted as interface rankings. Full numerical pairwise estimates are provided in Supplementary Material 4. Reliability/information quality scores across interfaces. Values are median [IQR]. Within each row, values sharing at least one superscript letter did not differ significantly; values with no superscript letter in common differed significantly in paired Wilcoxon signed-rank tests after Benjamini–Hochberg adjustment. Overall p-values were obtained using Friedman tests, and Kendall’s W is reported with its bootstrap 95% CI. Letters are grouping labels only and should not be interpreted as interface rankings. Full numerical pairwise estimates are provided in Supplementary Material 4. Readability All six readability indices differed significantly across interfaces, although the pairwise patterns and overall effect sizes varied by metric. Kendall’s W ranged from 0.064 to 0.389, indicating that the magnitude of between-interface differences was not uniform across readability measures. Accordingly, the readability findings were metric-specific and should not be interpreted as establishing a single overall interface ranking. The distributions of readability scores are shown in Figure 4 and Table 4 summarizes the descriptive statistics, overall comparisons, effect sizes, and Benjamini–Hochberg-adjusted pairwise groupings. Complete numerical pairwise estimates, 95% confidence intervals, and adjusted p values are provided in Supplementary Material 4. Distributions of readability scores across interfaces. Table 4Readability scores across interfaces.MetricChatGPTGeminiCopilotDeepSeekDoubaoQwenOverall pKendall’s WKendall’s W 95% CIARI13.97 ± 3.77a14.24 ± 2.72a14.73 ± 2.57a15.15 ± 4.44a22.66 ± 14.19b13.95 ± 2.14a0.0020.0720.033–0.169CL12.45 ± 1.83a13.48 ± 1.96b14.65 ± 1.48c12.69 ± 1.82a14.82 ± 2.06c15.00 ± 1.68c<0.0010.3890.304–0.504FKGL12.91 ± 2.93a13.59 ± 2.44b13.86 ± 2.25b14.13 ± 3.57b19.92 ± 11.41c13.60 ± 1.92b0.0050.0640.024–0.163GFI15.86 ± 3.06a16.54 ± 2.67b17.09 ± 2.62b17.22 ± 3.82b23.17 ± 12.01c16.80 ± 2.14b0.0040.0650.027–0.172SMOG14.11 ± 1.79a14.80 ± 1.90b15.27 ± 1.87b15.03 ± 2.39b18.32 ± 6.53c14.95 ± 1.44b<0.0010.0910.048–0.189FRES44.17 ± 10.83a37.18 ± 12.99b33.53 ± 10.33c39.10 ± 12.68b18.63 ± 34.39d31.07 ± 10.25cd<0.0010.2410.168–0.355Higher FRES values indicate easier readability, whereas lower ARI, CL, FKGL, GFI, and SMOG values indicate easier readability. Within each row, values sharing at least one superscript letter did not differ significantly; values with no superscript letter in common differed significantly in paired Wilcoxon signed-rank tests after Benjamini–Hochberg adjustment. Overall p-values were obtained using Friedman tests, and Kendall’s W is reported with its bootstrap 95% CI. Letters are grouping labels only and should not be interpreted as interface rankings. Full numerical pairwise estimates are provided in Supplementary Material 4. Readability scores across interfaces. Higher FRES values indicate easier readability, whereas lower ARI, CL, FKGL, GFI, and SMOG values indicate easier readability. Within each row, values sharing at least one superscript letter did not differ significantly; values with no superscript letter in common differed significantly in paired Wilcoxon signed-rank tests after Benjamini–Hochberg adjustment. Overall p-values were obtained using Friedman tests, and Kendall’s W is reported with its bootstrap 95% CI. Letters are grouping labels only and should not be interpreted as interface rankings. Full numerical pairwise estimates are provided in Supplementary Material 4. Discussion Main findings In this comparative evaluation of six publicly accessible search-enabled LLM interfaces for mpox-related public health consultation, unsafe responses were relatively infrequent and did not differ significantly across interfaces after correction for multiple comparisons; nevertheless, potentially harmful outputs occurred in every interface. Significant between-interface differences were observed for Accuracy, Empathy, reliability/information quality, and readability, although the magnitude and pairwise patterns varied by measure. These findings represent interface-level performance within the present response set and recorded query conditions rather than evidence of intrinsic or stable superiority. Mpox consultation is particularly demanding because responses must integrate symptom recognition, transmission risk, post-exposure action, vaccination, care seeking, jurisdiction-specific guidance, uncertainty, and non-stigmatizing communication (1, 5, 8–10, 33). The identified safety concerns—including unsupported local-risk estimates, over-reassurance, ambiguous post-exposure advice, overstated vaccine or treatment effects, and stigma-sensitive framing—illustrate why fluent answers may still be incomplete or unsafe. In the present mpox-specific context, search-enabled LLM interfaces may warrant further evaluation for a possible adjunctive role in mpox-related information delivery; however, the present response-level evaluation does not establish their effectiveness, acceptability, or influence on user behavior. These findings support disease-specific evaluation, ongoing monitoring, and appropriate safeguards before broader use in mpox consultation (15, 17). Safety, accuracy, and empathy Safe-response rates were numerically high across the evaluated interfaces, but neither the overall Safety comparison nor any Benjamini–Hochberg-adjusted pairwise comparison was statistically significant. The observed differences in percentages should therefore be interpreted descriptively rather than as evidence of differential safety performance. Nevertheless, every interface produced responses meeting the prespecified majority-vote unsafe criterion. Among the 31 unsafe responses, recurrent concerns involved unverified jurisdiction-specific information, transmission or contaminated-material risks, post-exposure management, vaccination or treatment claims, atypical presentations, and stigmatizing communication. The exploratory reassessment further showed variation in potential-harm severity. Thus, the binary Safety outcome identifies whether a response met the unsafe criterion but does not fully convey the type, severity, or plausible clinical and public-health consequences of the content, particularly in relation to exposure management, vaccination, isolation, and time-sensitive guidance (1, 40–42). The unsafe-response range observed in the present study (7.7–13.5%) broadly overlaps with the 5–13% reported by Draelos et al. in an evaluation of 888 responses to 222 patient-posed medical questions (43). Direct numerical comparison is limited by differences in clinical scope, question construction, safety definitions, evaluator composition, and query procedures. Nevertheless, both studies indicate that generally high safe-response rates do not eliminate potentially important safety failures. Accuracy showed clearer between-interface differentiation than binary Safety, although these findings should be interpreted as condition-specific interface-level patterns rather than intrinsic differences between the underlying base models or evidence of stable interface superiority. Previous studies have similarly shown that fluent patient-facing responses may still be incomplete, insufficiently qualified, or unsupported (12–14, 16). In contrast to the present findings, a guideline-based evaluation of 27 hypothyroidism questions reported high overall accuracy across ChatGPT-3.5, DeepSeek-V3, and Gemini-2.5, without significant overall differences in Accuracy or DISCERN scores (44). The different patterns may partly reflect differences in disease context, question breadth, prompting strategy, search and retrieval conditions, interface versions, and the greater importance of time-sensitive, jurisdiction-dependent, and exposure-specific guidance in mpox consultation. Empathy also varied across interfaces. In mpox consultation, empathetic communication is clinically and publicly relevant because questions frequently involve sexual exposure, pregnancy, children, immunocompromise, privacy, and stigma. Responses should be supportive without minimizing risk and nonjudgmental without obscuring necessary clinical or public-health action. Although previous studies have reported favorable empathy ratings for some chatbot-generated responses (12, 20), the present findings indicate that supportive and non-stigmatizing communication was not uniform across the evaluated interfaces. Empathy, privacy protection, and stigma avoidance should therefore be regarded as safety-relevant communication dimensions rather than optional stylistic features (40). Overall, these findings support multidimensional evaluation incorporating Safety, Accuracy, and communication quality rather than reliance on fluency or binary Safety alone. Reliability/information quality Reliability and information quality varied significantly across the evaluated interfaces, although the pairwise patterns differed by instrument. This indicates that apparently plausible and well-structured responses may still vary in evidence alignment, completeness, transparency, and practical usefulness. These findings should be interpreted as properties of the evaluated interfaces, retrieval settings, and query period rather than as stable characteristics of the underlying base models. Our findings parallel a recent public-health evaluation of chatbot-generated information on thunderstorm asthma, which used DISCERN, EQIP, JAMA benchmarks, GQS, and the same six readability indices and likewise identified instrument-dependent differences across interfaces (25). Together, these studies support multidimensional assessment rather than reliance on a single global quality score, because fluent or comprehensive responses may still lack source attribution, currency, uncertainty disclosure, or actionable guidance. JAMA scores were generally low, indicating limited transparency regarding authorship, attribution, disclosure, and currency (23). This is particularly relevant to mpox because current risk, vaccine eligibility, diagnostic access, travel advice, and public-health recommendations may vary across time and jurisdictions. The qualitative safety findings further showed that unclear source grounding or temporal applicability may contribute to misleading local-risk estimates, over-reassurance, and inappropriate exposure, vaccination, or treatment advice. Public-facing interfaces should therefore distinguish stable medical knowledge from changing public-health guidance, clearly communicate uncertainty and source dates, and direct users to authoritative services when recommendations are time- or jurisdiction-dependent. Future evaluations should continue to combine measures of reliability, patient-information quality, transparency, and actionability with qualitative safety analysis (21–23, 26). Readability Readability differed significantly across the evaluated interfaces, although the effect sizes and pairwise patterns varied by index. The findings were therefore metric-specific and should not be interpreted as establishing a single overall readability ranking. This variation remains relevant because users may need to make timely decisions regarding care seeking, symptom monitoring, vaccination, exposure precautions, and contact with public-health services. Even factually accurate information may have limited practical value if it is text-dense, jargon-heavy, or difficult to translate into action. Public-facing responses should therefore prioritize plain language, concise sentences, explicit action steps, and clear differentiation between urgent and non-urgent situations (18, 19). Readability formulas primarily capture surface-level textual characteristics, such as sentence length, word length, syllables, and characters, rather than actual comprehension, trust, emotional response, behavior, or medical correctness (27–32). Moreover, previous studies suggest that readability patterns may vary by clinical topic, interface version, query settings, and prompt design (25, 44–46). Future research should therefore combine formula-based indices with direct testing among public users, particularly those with lower health literacy, high anxiety following possible exposure, or limited access to professional health advice. Practical implications Within the context of mpox-related public health consultation, clinicians should recognize that patients may seek or present with chatbot-generated information and should verify exposure history, symptoms, vaccination status, and jurisdiction-specific recommendations rather than assuming that the generated advice is accurate or current. Public-health professionals may consider periodically reviewing the mpox-related information provided by widely used interfaces, maintaining accessible and frequently updated official guidance, and encouraging clear presentation of source dates, geographic applicability, uncertainty, and escalation pathways. For members of the public, any mpox information obtained from search-enabled LLM interfaces should be treated as preliminary, verified against qualified clinical or public-health sources, and not used as a substitute for professional assessment, particularly after a relevant exposure, when new lesions or systemic symptoms occur, or for pregnant, pediatric, or immunocompromised individuals. Advice concerning testing, isolation, vaccination, treatment, and travel should be confirmed through local health authorities or qualified clinicians. In mpox-related public-facing communication, responses should use plain language, concise action steps, and non-stigmatizing wording (8–10, 18, 19, 40). Limitations This study has several limitations. First, the question set was investigator-developed through purposive multi-source identification rather than sampled from a validated or probabilistically representative public-question dataset. Although the source platforms, screening criteria, agreement results, and item-level decisions were documented, prioritization within the fixed 52-question framework required investigator judgment. Other researchers might therefore derive a different question set, and the present benchmark should be interpreted as a structured, coverage-oriented sample rather than a representative sample of all public mpox queries. Second, explicit interface identifiers were removed from the rating materials, but the six responses within each question block were not randomized, and the effectiveness of identity masking was not formally assessed. Raters may have inferred interface identity from linguistic style, response structure, or source-presentation patterns, and the fixed order may have introduced position or expectation effects. High inter-rater agreement indicates scoring consistency but does not exclude shared systematic bias. Additionally, the majority-vote rule reduced five individual judgments to a binary response-level classification. Safety concerns raised by only one or two raters were therefore not captured in the primary Safety outcome, and the reported safe-response rates should not be interpreted as unanimous agreement, although complete de-identified rater-level ratings are available in Supplementary Material 6. Third, each question was submitted only once to each interface during a predefined query period. Search-enabled outputs may vary across sessions because of stochastic generation, retrieved sources, interface updates, geographic location, and time. The findings therefore represent a single-query cross-sectional snapshot and do not establish session-level reproducibility, temporal stability, or stable between-interface rankings. Fourth, the severity and thematic analyses were post hoc and reflected expert judgments of plausible potential harm rather than observed user behavior or clinical outcomes. Actual consequences may vary according to health literacy, vulnerability, exposure context, and access to care. Because the same raters conducted both the original binary assessment and the exploratory reassessment, prior familiarity with some responses and residual inference of interface identity cannot be excluded. Fifth, all interfaces were evaluated with web-search or intelligent-search functions enabled. Consequently, the observed performance cannot be attributed solely to the underlying language models and may also reflect retrieval quality, source selection, search ranking, and interface-level safety mechanisms. Sixth, the standardized prompts were evaluated in English, and the study relied on expert scoring and formula-based readability indices rather than direct testing with members of the public. The findings may therefore not fully generalize to other linguistic and cultural settings or capture actual comprehension, trust, emotional response, risk perception, and health-related behavior. In addition, DISCERN, EQIP, JAMA, and related measures evaluated the displayed responses as written consumer health information rather than the complete interactive or technical capabilities of the interfaces. Finally, the guideline-based reference framework was developed specifically for this study and was not an independently validated gold standard. Although authoritative guidelines and recommendations were prioritized and all entries underwent independent external expert review, the selection and interpretation of evidence and the formulation of expected public-facing answers involved expert judgment. This subjectivity may have influenced response scoring and the observed performance of the evaluated interfaces. Conclusion In this single-query comparative cross-sectional evaluation, the six publicly accessible search-enabled LLM interfaces showed heterogeneous response-level performance across Safety, Accuracy, Empathy, reliability/information quality, and readability in mpox-related public health consultation. Potentially harmful outputs were identified in every interface, particularly in time-sensitive, jurisdiction-specific, post-exposure, vaccination, treatment, contaminated-material, and stigma-sensitive scenarios. These findings underscore the need for guideline-based evaluation, source transparency, readability optimization, and robust safety safeguards. However, because public comprehension, trust, behavioral responses, care-seeking decisions, and clinical or public-health outcomes were not evaluated, this study does not establish the real-world effectiveness or suitability of these interfaces as public-facing information tools. Any potential adjunctive role in mpox communication requires prospective, user-centered evaluation. The findings represent a time- and configuration-specific interface-level snapshot under the recorded testing conditions and should not be interpreted as evidence of reproducible performance or consistent superiority of any particular interface across sessions, versions, or settings. Whether similar patterns occur in other infectious-disease or public-health contexts remains uncertain and should be examined in separate disease-specific studies.