MATE: Diagnosing Empathy Calibration Failures in Multi-Turn Human–LLM Interaction

Yeseon Hong, Junhyuk Choi, Minju Kim, Bugeun Kim · 2025

Read MATE: Diagnosing Empathy Calibration Failures in Multi-Turn Human–LLM Interaction by Yeseon Hong, Junhyuk Choi, Minju Kim, Bugeun Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24658.txt.

Introduction

Yeseon Hong and co-authors build a test frame in a 2025 paper for how chat models show empathy over many turns. As models step into tender talks, one warm reply is not enough. Warmth can drift, ring hollow, or swell into mere nodding as a chat runs on.

Their frame grades empathy as a flow shaped by what the user bares, turn by turn. In two studies with more than eighty people, stock replies missed the mark by flat, stock warmth. A self-check prompt eased that flaw but bred a new one: too much agreement. Chat builders gain a lens that sees empathy fail in motion, not just in one shot.

Catalog record

CREATORYeseon Hong, Junhyuk Choi, Minju Kim, Bugeun Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24658

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MATE: Diagnosing Empathy Calibration Failures in Multi-Turn Human–LLM Interaction Yeseon Hong, Junhyuk Choi, Minju Kim, Bugeun Kim Abstract Large language models are increasingly used in emotionally consequential interactions. Response-level evaluation, however, struggles to diagnose how empathy fails across turns. We introduce MATE ( M ulti-turn A ssessment of calibra T ed E mpathy), a framework that evaluates empathy as a multi-turn, perception-centered process. Across two controlled studies ( N = 82 N=82 ), baseline responses exhibit disclosure-insensitive miscalibration: generic validation and formulaic tone read as hollow across turns. A prompt-level self-critique condition makes these patterns less prominent but coincides with a different failure, agreement-skewed miscalibration, where affirmation becomes insufficiently contingent on disclosure context under high persona alignment (62.5% of fit participants in Study 2). Response-level analysis is consistent with a tone–agreement dissociation, as the condition shifts tone toward naturalness while agreement density also increases (+98% under fit vs. +25% under unfit). These findings reframe empathic behavior as relational calibration—consistency among disclosure depth, persona, and response behavior—rather than maximization. Response behavior itself spans tone, agreement, and contextual specificity, sub-dimensions that can drift independently. 2 2 footnotetext: Corresponding author 1 Introduction As large language models (LLMs) become embedded in everyday life, LLM-based chatbots are increasingly used in emotionally consequential interactions. Users turn to LLM-based chatbots for comfort and to discuss personal or emotionally difficult experiences ( Maples et al., 2024 ; Luo et al., 2025 ; Pentina et al., 2023 ) . In such settings, empathy becomes central to human–LLM emotional interaction, and the NLP and Human–AI Interaction (HAI) communities have actively studied how to generate and evaluate empathic responses from LLMs ( Rashkin et al., 2019 ; Lin et al., 2019 ; Sharma et al., 2020 ) . Much of this work has treated empathy primarily as a property of responses, asking whether LLM replies are empathic or emotionally attuned to users ( Liu et al., 2021 ; Welivita et al., 2021 ; Gao et al., 2021 ) . Yet emotional interaction unfolds across sustained conversation rather than isolated replies. This response-centered view therefore leaves two aspects of empathy in human–LLM interaction underexamined: how empathy develops across turns, and how users perceive it as the conversation unfolds. First, empathy develops across turns rather than within isolated replies. Psychological work treats empathy as a process that unfolds through ongoing disclosure and responsive engagement ( Reis and Shaver, 1988 ; Laurenceau et al., 1998 ) , and related work on sycophancy shows that over-validation tendencies compound across turns in ways that response-level evaluation cannot diagnose ( Sharma et al., 2024 ; Hong et al., 2025 ; Cheng et al., 2025 ) . Second, what models express and what users perceive can diverge. Linguistic markers of empathy do not reliably predict whether users feel understood, and chatbots often rely on formulaic acknowledgments that engage poorly with the specifics of user disclosures ( Xu and Jiang, 2024 ; Cuadra et al., 2024 ; Liu et al., 2025 ; Chen et al., 2024 ) . Beyond these substantive findings, existing empathy measures are themselves poorly suited for capturing either aspect. Trait-level scales assess stable dispositions ( Davis, 1983 ; Baron-Cohen and Wheelwright, 2004 ) and response-level annotations evaluate isolated utterances ( Alam et al., 2018 ) , but neither tracks how perceived empathy shifts as conversational context accumulates. Addressing both gaps requires a framework that treats empathy as something that unfolds between partners over time. Human–human interaction research provides such a foundation. Interpersonal process models hold that empathy develops through repeated cycles of self-disclosure and responsive engagement, and that what feels empathic is shaped by how responses fit the disclosures they follow ( Reis and Shaver, 1988 ; Laurenceau et al., 1998 ) . This perspective connects both aspects at once: disclosure-response cycles are the unit through which empathy accumulates across turns, and the disclosures themselves provide the conversational context against which responses are perceived. Motivated by this perspective, we adopt reciprocal self-disclosure—where both partners share personal information at comparable depth—as our interactional foundation. We introduce MATE ( M ulti-turn A ssessment of calibra T ed E mpathy), an interaction framework that operationalizes this perspective. Following interpersonal process theory, MATE treats empathy as processual rather than scalar: empathy operates through cycles of disclosure and responsive engagement, and its success or failure is reflected in how users perceive responses and in the relational outcomes those responses shape. Its three components are staged reciprocal self-disclosure, repeated assessment of user perception, and context-bound interpretation of experienced responses. Applying MATE, we conduct two controlled studies ( N=82 ). Study 1 characterizes how empathy is perceived under standard LLM responses, identifying failure modes that surface only when perception accumulates across stages. Study 2 uses prompt-level self-critique as a diagnostic probe, examining how a tone-oriented revision condition shifts the patterns observed in Study 1. Our main contributions are as follows. (1) We introduce MATE, a framework that evaluates empathy as a multi-turn, perception-centered phenomenon by integrating staged reciprocal self-disclosure, repeated perception assessment, and context-bound qualitative analysis. (2) Applying MATE, we identify failure modes that response-level evaluation cannot surface: disclosure-insensitive miscalibration in baseline responses, where generic validation reads as hollow despite linguistic markers of empathic engagement; and agreement-skewed miscalibration under high persona alignment, where naturalness gains coincide with affirmation that is insufficiently contingent on the disclosure context. (3) These findings support reframing empathic LLM behavior as a problem of relational calibration rather than a maximization objective, and reveal a tone–agreement dissociation invisible to single-response evaluation. 2 Related Work 2.1 Response-Centered Empathy Evaluation in Dialogue Systems Empathy in dialogue systems has been studied primarily through datasets and architectures targeting empathic response generation. Following EmpatheticDialogues ( Rashkin et al., 2019 ) , subsequent work expanded this line through architectural designs such as mixture-of-experts listeners ( Lin et al., 2019 ) and emotion-cause modeling ( Gao et al., 2021 ) , and through emotional support datasets such as ESConv ( Liu et al., 2021 ) . These works evaluate empathy at the response level, judging whether a single reply bears linguistic and affective markers of empathic engagement. Two recent lines suggest that response-level evaluation misses important aspects of empathy in interaction. Research on sycophancy shows that LLMs trained with human feedback systematically over-validate users, and that this tendency compounds across turns rather than manifesting within any single response ( Sharma et al., 2024 ; Hong et al., 2025 ; Cheng et al., 2025 ) —a cross-turn accumulation pattern that response-level evaluation cannot diagnose. Work on perceived empathy further shows that expressed empathic markers do not reliably predict whether users feel understood ( Xu and Jiang, 2024 ; Liu et al., 2025 ) , and that LLMs recognize emotional content yet often fail to engage with users’ specific disclosures in contextually grounded ways ( Cuadra et al., 2024 ; Chen et al., 2024 ) . Together, these findings point to multi-turn dynamics and users’ perception as two dimensions that response-level evaluation leaves underexamined, motivating a framework that addresses both. 2.2 Reciprocal Self-Disclosure and the Development of Perceived Empathy Interpersonal process theory provides a foundation for evaluating empathy as a multi-turn, perception-centered process. Empathy is not located in individual responses but emerges through cycles in which one partner

Read the complete text