Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI
Agnik Saha, Victoria Churchill, Anny D. Rodriguez, Ugur Kursuncu, Muhammed Y. Idris · 2025
Read Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI by Agnik Saha, Victoria Churchill, Anny D. Rodriguez, Ugur Kursuncu, Muhammed Y. Idris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10472.txt.
Introduction
Agnik Saha and colleagues graded language models as cancer communicators in a 2025 study. Public understanding of breast and cervical cancer lags on prevention, screening, and treatment, and chatbots are already filling the gap with uneven quality. Eight models, five general-purpose and three medical, were scored in a mixed-methods framework across language quality, safety and trustworthiness, and accessibility of communication, combining metrics, expert ratings, and statistical tests.
General models wrote better and connected more effectively, while medical models communicated more accessibly. Worryingly, the medical models also showed more potential harm, toxicity, and bias, dragging their safety scores down. Specialization, the authors conclude, does not guarantee safer health communication.
Catalog record
| CREATOR | Agnik Saha, Victoria Churchill, Anny D. Rodriguez, Ugur Kursuncu, Muhammed Y. Idris |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-10472
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI Agnik Saha, Victoria Churchill, Anny D. Rodriguez, Ugur Kursuncu, Muhammed Y. Idris Abstract Effective communication about breast and cervical cancers remains a persistent health challenge, with significant gaps in public understanding of cancer prevention, screening, and treatment, potentially leading to delayed diagnoses and inadequate treatments. This study evaluates the capabilities and limitations of Large Language Models (LLMs) in generating accurate, safe, and accessible cancer-related information to support patient understanding. We evaluated five general-purpose and three medical LLMs using a mixed-methods evaluation framework across linguistic quality, safety and trustworthiness, and communication accessibility and affectiveness. Our approach utilized quantitative metrics, qualitative expert ratings, and statistical analysis using Welch’s ANOVA, Games-Howell, Hedges’ g. Our results show that general-purpose LLMs produced outputs of higher linguistic quality and affectiveness, while medical LLMs demonstrate greater communication accessibility. However, medical LLMs tend to exhibit higher levels of potential harm, toxicity, and bias, reducing their performance in safety and trustworthiness. Our findings indicate a duality between domain-specific knowledge and safety in health communications. The results highlight the need for intentional model design with targeted improvements, particularly in mitigating harm and bias, and improving safety and affectiveness. This study provides a comprehensive evaluation of LLMs for cancer communication, providing critical insights for improving AI-generated health content and informing future development of accurate, safe, and accessible digital health tools. keywords Large Language Models (LLMs), Generative AI, Health Communications, Breast Cancer, Cervical Cancer, Public Health, Human-AI Interaction 1 Introduction Cancer remains a leading cause of morbidity and mortality among women in the U.S., making it a critical public health issue. One in three women in the U.S. will be diagnosed with cancer in her lifetime, and one in six will die from the disease. Breast cancer is the most commonly diagnosed cancer among women, with an estimated 310,720 new cases and 42,250 deaths projected in 2024 ( American Cancer Society, (2023) ) . Despite improvements in screening and treatment, disparities in cancer outcomes persist. For instance, Black women experience a 40% higher breast cancer mortality rate than White women, despite similar incidence rates, largely due to systemic inequities in screening access, delayed diagnoses, and unequal healthcare ( Siegel et al., (2023) ; Warner et al., (2012) ; Moore et al., (2023) ) . Cervical cancer, the most common cancer among women in the U.S., had 13,360 new cases in 2025 ( American Cancer Society, (2024) ) , with Black women facing a mortality rate 200% higher than White women and Hispanic women experiencing a 51% higher incidence rate ( Olusola et al., (2019) ; Moore de Peralta et al., (2017) ; Spencer et al., (2023) ) . These disparities are rooted in broader structural and social barriers, including financial hardship, limited geographic access, and psychological challenges ( Consedine et al., (2004) ) . Emerging technologies, such as Large Language Models (LLMs), offer promising generative AI applications to support potentially more effective and equitable health communications for breast and cervical cancers. LLMs have the potential to improve access to reliable, understandable, and personalized health information. However, their rapid advancement has outpaced research into their real-world effectiveness and safety in clinical and public health settings. Existing studies provide limited evaluation of how well LLMs deliver cancer-related information that is accurate, unbiased, and accessible ( Johnson et al., (2016) ; Grilo et al., (2025) ) . Experts strongly encourage that before we commit to this new frontier in cancer communications, we need to address accuracy, safety, and privacy ( Zitu et al., (2025) ) . These identified gaps are particularly concerning given the risks of misleading or incorrect information, which can delay diagnoses, influence harmful treatment decisions, and erode trust in health institutions ( Swire-Thompson et al., (2020) ) . Given the urgency of addressing the need for high-quality communication tools in cancer care, this study was conducted to evaluate the quality and safety of LLM-generated content in the context of breast and cervical cancer. Our rationale stems from the critical importance of ensuring that AI-driven tools do not exacerbate existing disparities or contribute to patient harm. The specific objectives of this work are two-fold: • Develop a comprehensive evaluation framework that systematically assesses LLMs across three dimensions: linguistic quality, safety and trustworthiness, and communication accessibility and affectiveness; • Characterize the relative strengths and limitations of general-purpose and specialized medical LLMs for cancer communications. In this study, we developed an evaluation framework for effective cancer communication with quantitative and qualitative elements, which span three main categories: (i) Linguistic Quality, (ii) Safety & Trustworthiness, and (iii) Communication Accessibility & Affectiveness. Five general‑purpose LLMs (Llama 3, Gemma, Alpaca, Mistral, Vicuna) and three specialized medical LLMs (MedAlpaca, BioMistral, Meditron) were utilized for evaluation. For linguistic quality, we used quantitative metrics for semantic coherence and consistency, and hallucinations, as well as qualitative ratings by domain experts assessing accuracy, coherence, jargon control, and reasoning. Safety and trustworthiness were assessed using quantitative toxicity and bias metrics, complemented by expert evaluations of harm and user trust. Finally, communication accessibility and affective impact were measured with readability indices, reflection score, and qualitative assessments of clarity, empathy, compassion, actionability, domain relevance, and user acceptability. Our findings show that general-purpose LLMs, particularly Llama 3 and Gemma, outperformed specialized medical models in Linguistic Quality. In contrast, medical LLMs, such as MedAlpaca and BioMistral demonstrated better Communication Accessibility, generating outputs with lower grade-level requirements and reduced complexity. While general-purpose LLMs, especially Llama 3, demonstrated more affective communication, general-purpose models, such as Alpaca and Llama 3, consistently outperformed medical LLMs in Safety and Trustworthiness, producing responses that were safer, less toxic, and less biased. 2 Related Work Recent research has increasingly focused on evaluating LLMs in healthcare across both general-purpose and domain-specific settings. A systematic review identified over 760 LLM evaluations, with more than 93% involving general-purpose models such as GPT-3.5 and GPT-4, and fewer than 6% assessing specialized medical LLMs ( Shool et al., (2025) ) . Google’s Med-PaLM series achieved 86.5% on USMLE-style exams and outperformed clinicians on eight of nine physician preference axes ( Singhal et al., (2025) ) . Other proprietary models, such as Claude, have also been assessed along with the GPT family of LLMs; in oncology-focused evaluations, the responses from GPT-3.5, GPT-4, and Claude were rated higher than those from physicians in empathy, quality, and readability ( D. Chen et al., (2024) ) . Another comparative analysis in the breast cancer context showed that GPT-4 outperformed GPT-3.5 and Claude 2 in accuracy, relevance, and clinical applicability across assessment, treatment planning, and psychosocial care ( Deng et al., (2024) ) . In oncology, LLM applications have spanned patient education, triage, and physician note drafting, with nearly half of oncology studies focusing on patient communication, followed by diagnostic support (20%) and treatment recommendations (17%) ( D. Chen et al., (2025) ) . Studies such as Chen et al. used panels of oncologists and LLMs (GPT-3.5, GPT-4, Claude) to evaluate responses to real-world patient questions, underscoring LLMs’ potential as virtual cancer consultants ( D. Chen et al., (2024) ) . In primary care, evaluations have included patient forum moderation ( Ayers et al., (2023) ) , licensing exam simulations