When Seekers Are Hard to Help: Evaluating Emotional Support Dialogue Systems in Worst-Case Interactions

Jiajie Yang, Yangchun Li, Guanyi Chen, Rui Fan, Xin Bai, and Tingting He · 2026

Read When Seekers Are Hard to Help: Evaluating Emotional Support Dialogue Systems in Worst-Case Interactions by Jiajie Yang, Yangchun Li, Guanyi Chen, Rui Fan, Xin Bai, and Tingting He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28228.txt.

Introduction

Yang and colleagues test emotional-support chatbots against users who are hard to help: tuned out, pushy back, or grimly fixed. With expert-played hard cases and a worst-case test frame, nearly all seventeen systems fell far, though big general models held up best. Hard-case fake chats also toughened small models in training.

Catalog record

CREATORJiajie Yang, Yangchun Li, Guanyi Chen, Rui Fan, Xin Bai, and Tingting He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70285
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28228

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Seekers Are Hard to Help: Evaluating Emotional Support Dialogue Systems in Worst-Case Interactions Jiajie Yang, Yangchun Li, Guanyi Chen, Rui Fan, Xin Bai, and Tingting He Abstract Emotional Support Dialogue Systems (ESDSes) are increasingly evaluated and trained with LLM-simulated seekers. However, such simulated seekers often behave as cooperative, average-case users who disclose clearly, respond constructively, and accept support within a few turns. This can lead to overly optimistic evaluation and obscure whether ESDSes can handle difficult help-seeking interactions. In this work, we study ESDS evaluation under worst-case interactions, where seekers are hard to help due to low engagement, resistance, limited self-disclosure, emotional volatility, or rigid negative interpretations. We first conduct an expert simulation study with eight experienced counselling professionals, who simulate difficult seekers, interact with existing Chinese ESDSes, provide scale ratings, and participate in semi-structured interviews. Based on this study, we derive worst-case seeker behaviours and identify key limitations of current systems. We then propose a worst-case evaluation framework consisting of an LLM-based worst-case seeker simulator and four worst-case-oriented metrics: Deep Emotional Understanding, Guided Exploration, Balanced Emotional Support, and Authentic and Grounded Support. Evaluating 17 systems, we find that nearly all models suffer substantial performance drops under worst-case interactions. Large general-purpose LLMs are generally more robust than specialised ESDSes, but even the strongest models struggle to sustain engagement and improve seekers’ emotional states. Finally, we show that worst-case simulation can also generate useful training data, improving the robustness of smaller models. 1 Introduction Emotional Support Dialogue Systems (ESDSes) aim to provide empathetic, supportive, and contextually appropriate responses to users experiencing emotional distress ( Liu et al. 2021 ) . With the rapid development of large language models (LLMs), such systems have become increasingly fluent and accessible ( Kang et al. 2024 ; Team 2024 ; Qiu and Lan 2024 ; Qiu et al. 2024a ; Zhang et al. 2024 ; Xie et al. 2025 ; Bai et al. 2025 ; Bai et al. 2026 ) . This makes reliable evaluation crucial: emotional support systems should not only generate plausible responses, but also provide robust support across diverse help-seeking situations. Figure 1: Average-case vs. worst-case supports. Recent emotional support dialogue research increasingly relies on LLM-simulated seekers for both training ( Qiu and Lan 2024 ; Ye et al. 2025 ) and evaluation ( Zhao et al. 2024 ; Ye et al. 2026 ) . These simulation-based protocols enable scalable data generation and system assessment, but they may inherit a key limitation of LLM-based social simulation ( Dillion et al. 2023 ; Schmidt et al. 2024 ; Cao et al. 2025 ) : unless explicitly controlled, LLM agents tend to produce average, normative, and socially cooperative behaviours. In emotional support settings, this can lead to simulated seekers who clearly disclose their problems, respond constructively to questions, accept reframing, and become reassured within only a few turns. As illustrated in Figure 1, such interactions may be smooth and easy for support systems to handle, creating an average-case bias in evaluation. This bias is problematic because difficult help-seeking interactions are not peripheral edge cases. They represent situations in which support systems are most likely to fail, and in which users may particularly need patient, adaptive, and robust support. In real support conversations, seekers may respond minimally, reject suggestions, distrust the system, avoid the central issue, or remain ambivalent about receiving help. A system that appears empathetic with cooperative seekers may still fail in these worst-case interactions by giving premature advice, repeating generic validation, over-questioning the seeker, or failing to repair conversational breakdowns. We argue that ESDSes should be evaluated not only on cooperative, average-case seekers, but also on worst-case interactions where seekers are hard to help. Such evaluation can reveal robustness failures that conventional emotional support metrics may overlook, including poor engagement recovery, inability to handle distrust, lack of strategy adaptation, and failure to repair ruptures in the conversation. Figure 2: Overview of our worst-case evaluation framework. In this paper, we propose a worst-case evaluation framework for ESDSes, with the overall pipeline shown in Figure 2. We first conduct an expert simulation study with experienced counselling professionals, who simulate difficult help-seeking behaviours based on cases encountered in practice, interact with three mainstream support systems in Chinese, rate the conversations using existing emotional support scales, and participate in semi-structured interviews. From this study, we derive a taxonomy of worst-case seeker behaviours and identify failure modes of current LLM-based support systems. We then translate these findings into robustness-oriented evaluation metrics and an LLM-based worst-case seeker simulator for scalable stress testing. Our contributions are threefold. First, we conduct an expert simulation study to empirically characterise worst-case interactions in emotional support dialogue. Second, we derive a taxonomy of difficult seeker behaviours and robustness-oriented metrics for evaluating support systems under difficult interactions. Third, we develop an LLM-based worst-case seeker simulator and use it to stress-test emotional support systems, revealing robustness gaps and dimension-specific weaknesses that are not fully captured by conventional average-case evaluation. The code and data used in this study are public available at: https://github.com/YangJJ66/Worst-case-evaluation . 2 Expert Simulation Study To empirically ground the notion of worst-case interactions, we conduct an expert simulation study with experienced counselling professionals. Rather than defining difficult seekers solely through intuition or prompt engineering, we ask experts to simulate challenging help-seeking behaviours based on their professional experience, interact with existing emotional support systems, rate the resulting conversations using conventional emotional support scales, and reflect on system failures through semi-structured interviews. This study serves two purposes: first, to characterise what kinds of seeker behaviours make emotional support interactions difficult; and second, to identify the limitations of current LLM-based support systems when facing such interactions. 2.1 Study Design Participants and Evaluated Systems. Eight experienced psychological counsellors with doctoral-level training in psychology and at least 3 years of counselling experience were recruited for the study. Each expert was paid 100 RMB for the experiment. All sessions were conducted in person in a laboratory setting. We evaluated a broad set of Chinese ESDSes, including EmoLLM ( Team 2024 ) , MindChat ( Xin and Dong 2023 ) , Simpsybot_q ( Qiu and Lan 2024 ) , PsyChat ( Qiu et al. 2024b ) , CPsyCounX ( Zhang et al. 2024 ) , SoulChat2.0 ( Xie et al. 2025 ) , and MeChat ( Qiu et al. 2024a ) . In addition, we included Doubao Seed 2.0 Pro, a widely used general-purpose LLM in China. This choice was informed by a preliminary survey of 500 respondents conducted as part of this study, in which Doubao was reported as the most frequently used AI assistant. Procedure. Each expert interacted with three ESDSes through a self-developed counselling interface. System assignment was balanced such that each of the eight systems was evaluated by three experts. The order of systems was randomised, and system identities were hidden from participants. In each session, given a counselling situation (see Section 4.3 ), participants simulated a difficult seeker based on their counselling experience. To increase behavioural diversity, we also asked experts to vary the types of difficult seekers they simulated across the three conversations. Each conversation continued until the expert chose to end it, with an upper bound of 100 turns. After each conversation,

Read the complete text