Can “AI” be a Doctor? A Study of Empathy, Readability, and Alignment in Clinical LLMs
Mariano Barone, Francesco Di Serio, Roberto Moio, Marco Postiglione, Giuseppe Riccio, Antonio Romano, Vincenzo Moscato · 2026
Read Can “AI” be a Doctor? A Study of Empathy, Readability, and Alignment in Clinical LLMs by Mariano Barone, Francesco Di Serio, Roberto Moio, Marco Postiglione, Giuseppe Riccio, Antonio Romano, Vincenzo Moscato (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20791.txt.
Introduction
Barone, Di Serio, Moio, Postiglione, Riccio, Romano, and Moscato conduct a broad review of general and niche large language models. Base models amplify feeling slant compared with doctors, with Very Negative at about 43 to 45 percent versus about 37 percent. In larger designs such as GPT-5 and Claude, they produce harder language. Empathy-led prompting cuts extreme negativity and lowers grade level for GPT-5 but does not raise meaning match. Rephrase configurations achieve the highest semantic likeness to physician answers, up to about 93 percent, while raising readability and reducing feeling extremity.
Catalog record
| CREATOR | Mariano Barone, Francesco Di Serio, Roberto Moio, Marco Postiglione, Giuseppe Riccio, Antonio Romano, Vincenzo Moscato |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91664 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20791
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can “AI” be a Doctor? A Study of Empathy, Readability, and Alignment in Clinical LLMs Mariano Barone, Francesco Di Serio, Roberto Moio, Marco Postiglione, Giuseppe Riccio, Antonio Romano, Vincenzo Moscato Abstract Large Language Models (LLMs) are increasingly deployed in healthcare, yet their communicative alignment with clinical standards remains insufficiently quantified. We conduct a multidimensional evaluation of general-purpose and domain-specialized LLMs across structured medical explanations and real-world physician–patient interactions, analyzing semantic fidelity, readability, and affective resonance. Baseline models amplify affective polarity relative to physicians (Very Negative: 43.14–45.10% vs. 37.25%) and, in larger architectures such as GPT-5 and Claude, produce substantially higher linguistic complexity (FKGL up to 16.91–17.60 vs. 11.47–12.50 in physician-authored responses). Empathy-oriented prompting reduces extreme negativity and lowers grade-level complexity (up to -6.87 FKGL points for GPT-5) but does not significantly increase semantic fidelity. Collaborative rewriting yields the strongest overall alignment. Rephrase configurations achieve the highest semantic similarity to physician answers (up to μ = 0.93 \mu=0.93 ) while consistently improving readability and reducing affective extremity. Dual stakeholder evaluation shows that no model surpasses physicians on epistemic criteria, whereas patients consistently prefer rewritten variants for clarity and emotional tone. These findings suggest that LLMs function most effectively as collaborative communication enhancers rather than replacements for clinical expertise. keywords Large Language Models, Healthcare AI, Empathy in AI, Readability, Medical Communication, MedQuAD, Medical Question-Answering † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction As AI systems become increasingly capable and autonomous, ensuring their alignment with human values has become a practical concern rather than a purely theoretical one 25 ; 37 . In patient-facing medical applications, misalignment may result in unclear communication, inappropriate reassurance, or unsafe recommendations, with direct consequences for patient trust and clinical decision-making. This issue is particularly critical in healthcare, where Large Language Models (LLMs) are increasingly deployed in patient-facing settings, often in contexts characterized by vulnerability and emotional distress 6 . Recent evidence shows that over 150,000 clinicians across more than 150 institutions already rely on AI-powered systems to assist with patient messaging 1 1 1 https://www.nytimes.com/2024/09/24/health/ai-patient-messages-mychart.html , thus reshaping everyday clinical communication practices 21 ; 39 ; 35 . While existing research has largely focused on the factual accuracy of medical AI systems, particularly addressing hallucinations and reliability 18 ; 7 , other determinants of effective clinical communication remain comparatively underexplored. In this work, we focus on three communicative dimensions that are central in clinical interactions: semantic correctness (preservation of medical meaning), readability (linguistic accessibility to non-expert users), and affective appropriateness (alignment of emotional tone with patient needs). In the absence of these values, patients may experience confusion, anxiety, and erosion of trust in their healthcare providers 47 ; 22 ; 34 . Despite the existence of well-established communication frameworks such as SPIKES 9 and the Calgary-Cambridge Guide 26 , little empirical work has investigated whether modern LLMs reproduce or deviate from these communicative principles when interacting with patients 4 . We address this gap through a systematic evaluation of three communicative dimensions that are particularly relevant in clinical contexts: (1) semantic fidelity , how faithfully AI responses match expert clinical judgment; (2) readability , whether AI answers remain comprehensible across diverse literacy levels and cultural backgrounds; and (3) affective resonance , the extent to which AI responses acknowledge patients’ emotional needs. To this end, we conduct our analysis on a large-scale medical question-answering corpus comprising 47,457 entries derived from authoritative healthcare sources 1 , and investigate the following research questions: 1. RQ1 ( Empathy ): How do LLMs compare to human physicians in expressing empathy and emotional awareness in clinical communication? 2. RQ2 ( Readability ): Do LLM-generated responses differ from physician-authored answers in terms of linguistic readability? 3. RQ3 ( Prompt-based Alignment ): Does the use of empathy-oriented prompting improve the emotional tone and readability of LLM outputs while preserving semantic fidelity? 4. RQ4 ( Human-AI Collaboration ): Can LLMs improve the clarity and emotional appropriateness of physician-authored responses through collaborative rewriting? 5. RQ5 ( Expert-Patient Value Alignment ): To what extent do different LLM configurations satisfy the distinct preferences expressed by medical experts and patients? Our study addresses these questions through one of the first large-scale empirical comparisons between LLM-generated and physician-authored medical communication. We identify systematic differences between models and human clinicians in emotional tone, readability, and semantic consistency, with no single approach consistently outperforming the others across all dimensions. A central contribution of this work lies in the nature of the data examined. In contrast to prior studies that predominantly rely on social media content, patient self-reports, or synthetic dialogues, we analyze expert-authored medical communication produced by practicing clinicians in institutional settings. This enables an evaluation of alignment against high-standard clinical language as used in real healthcare workflows, rather than informal user-generated text. To the best of our knowledge, such data have been rarely leveraged in large-scale evaluations of LLMs for healthcare communication. 2 Related Work Empathy is widely recognized as a central component of effective clinical communication. In medical contexts, it is not merely a matter of emotional warmth but of calibrated emotional engagement, often described as detached concern 33 . Recent studies suggest that LLMs can generate emotionally resonant medical responses, although the empirical findings remain mixed. Ayers et al. 8 reported that 79% of Reddit AskDocs users preferred ChatGPT responses over those written by physicians, largely due to perceived empathy and tone. However, the informal context of online forums limits the generalizability of these findings to real clinical practice. More recently, analogous findings have been reported in oncological settings, where patients consistently rated chatbot responses as more empathetic than physician responses, further highlighting the divergence between lay and clinical perceptions of affective tone 14 . Similarly, Luo et al. 30 proposed EMRank as a metric for quantifying empathy in LLM responses, reporting higher empathy scores for ChatGPT compared to physicians, though their analysis focused primarily on emotional expression rather than overall clinical adequacy. Beyond emotional tone, readability and linguistic simplification represent further challenges in patient–provider communication. Roy et al. 38 showed that GPT-5 can improve comprehension of medical information; similar benefits have been demonstrated when AI is used to simplify surgical consent forms through human-AI collaborative approaches 5 , yet excessive simplification may reduce interpretability in complex clinical scenarios. A primary mechanism through which these dimensions are implicitly adjusted is prompt engineering. Prompt engineering has emerged as a key modulator of LLM behavior in medical settings. Prior work shows that techniques such as chain-of-thought prompting improve reasoning transparency and task performance 46 , while role conditioning can increase perceived empathy in generated responses. However, few studies have operationalized established clinical communication frameworks such as SPIKES 9 or the Calgary–Cambridge Guide 26 within prompt templates, limiting the transferability of these findings to structured clinical environments. These variations in prompt design not only affect how models generate responses, but also complicate direct comparisons with physician-authored communication. Several meta-analyses