CAREBench: Evaluating LLMs’ Emotion Understanding by Assessing Cognitive Appraisal Reasoning

Zhaoyue Sun, Hainiu Xu, Andero Uusberg, James J. Gross, Petr Slovak, Yulan He · 2026

Read CAREBench: Evaluating LLMs’ Emotion Understanding by Assessing Cognitive Appraisal Reasoning by Zhaoyue Sun, Hainiu Xu, Andero Uusberg, James J. Gross, Petr Slovak, Yulan He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17176.txt.

Introduction

Sun and colleagues introduce CAREBench, a benchmark that judges language models' grasp of emotion through the reasoning behind feelings, not just final labels. Built on appraisal theory, it annotates full inference chains from first- and third-person views on real stories. Tests across six models find strong systems match or beat human observers on some tasks yet lag on appraisal reasoning and positive emotions. Headline scores, the findings suggest, overstate true understanding.

Catalog record

CREATORZhaoyue Sun, Hainiu Xu, Andero Uusberg, James J. Gross, Petr Slovak, Yulan He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80577
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17176

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAREBench: Evaluating LLMs’ Emotion Understanding by Assessing Cognitive Appraisal Reasoning Zhaoyue Sun, Hainiu Xu, Andero Uusberg, James J. Gross, Petr Slovak, Yulan He Abstract Emotion understanding is a core capability for LLMs to interact effectively with humans, yet existing evaluation paradigms rely on discrete emotion label prediction and fail to capture the cognitive processes underlying emotion generation. Grounded in appraisal theory, we introduce CAREBench, the first benchmark with complete inferential chain annotations from both first- and third-person perspectives on real-world narratives, spanning appraisal reasoning, appraisal ratings, and multi-label emotion annotation. We propose a process-level evaluation framework and conduct systematic experiments across six LLMs organized around four research questions. We find that stronger models match or surpass human observers on certain tasks, yet fall short on appraisal reasoning and positive emotion recognition; performance across chain steps and sensitivity to appraisal interventions exhibit dissociations across models; and current models have not internalized the mechanisms needed to capture human subjective heterogeneity. These findings suggest that downstream emotion prediction metrics may overestimate LLMs’ true emotion understanding, and CAREBench provides a foundation for more diagnostically informative evaluation of LLMs’ affective cognitive capabilities. 1 Introduction Emotion understanding is a fundamental capability for AI systems built on Large Language Models (LLMs) deployed in affective computing, human-AI interaction, and mental health support. However, evaluating this capability remains challenging due to the complexity of how emotions are elicited, experienced, and expressed in humans. Most prior work has assessed emotion understanding through discrete emotion label prediction. However, this fails to capture the richness of real-world emotional phenomena 3 . To address this limitation, recent work has turned to psychological theories of emotion. Among these, appraisal theory posits that emotions arise from individuals’ subjective evaluations of events along multiple dimensions — such as goal relevance and coping potential — providing a structured and interpretable representation for evaluating emotion understanding 20 . Under this perspective, recent studies have introduced appraisal-grounded datasets to examine whether LLMs exhibit cognitive emotion understanding capabilities beyond label prediction 27 ; 11 ; 2 . However, existing appraisal-grounded datasets exhibit three key limitations. First, they predominantly treat appraisal dimension ratings as the sole representation of cognitive appraisal. However, much of appraisal involves inferences over mental states, such as beliefs and desires, while appraisal ratings reflect only its outcomes. Without explicitly modeling this inference process, it remains unclear whether a model truly understands the experiencer’s mental states or merely learns a superficial mapping from situations to ratings. Second, most datasets rely on single emotion category labels, whereas real-world emotional experiences are often mixed and involve multiple co-occurring emotions; thus, a single label fails to capture this richness. Third, the majority of datasets collect only third-person annotations, making it difficult to examine how perspective differences between experiencers and observers systematically influence emotion judgments. While the Crowd-Envent dataset 27 includes both first- and third-person perspectives, it contains very short situation descriptions (on average 20 words) and provides only appraisal dimension ratings. As a result, it cannot capture variation in annotators’ underlying appraisal inference processes. Figure 1 : Overview of the data annotation pipeline. To address these gaps, we constructed CAREBench, a benchmark designed to evaluate Cognitive Appraisal Reasoning in Emotion understanding, with three key features (Figure 1 ). First, we explicitly annotated the reasoning process of appraisals . Specifically, we recruited annotators to provide real-life emotional narratives (95 words on average), followed by three annotation layers collected in sequence: open-ended reasoning texts targeting five core appraisal dimensions, hierarchical appraisal ratings across 22 sub-dimensions, and emotion annotations. This design enables the construction of a situation → reasoning → appraisal → emotion inferential chain, making it possible to observe mental-state layer of appraisal, analyze its influence on fine-grained appraisal ratings, and evaluate LLMs’ emotion understanding correctness at the process level. Probing mental states before appraisal ratings also scaffolds the annotation process itself, ensuring that ratings are grounded in explicit reasoning rather than intuitive impression. To our knowledge, no existing dataset elicits mental states as a precursor to appraisal annotation, in either first-person or third-person settings. Second, for emotion annotation, instead of representing emotion as a single discrete label, we allow multi-label emotion annotation with intensity . This supports many-to-many analysis of appraisal–emotion mappings and more faithful modeling of complex real-world emotional responses. In addition, we provide full inferential-chain annotations from both the first-person experiencer and multiple third-person observers , enabling direct comparison between LLMs and both perspectives, as well as localization of subjective heterogeneity along the whole process. Annotator demographics and personality traits are also collected to support subjectivity analysis. Based on this dataset, we conducted systematic evaluations of six LLMs organized around four progressively structured research questions: how well models perform each inferential chain step, whether injecting appraisal reasoning helps with downstream prediction, whether models exhibit directional sensitivity to structured appraisal interventions, and whether they can capture the distributional heterogeneity of human appraisal judgments. Our key findings are: (1) although stronger models achieve competitive downstream performance, appraisal reasoning capabilities remain limited and positive emotion identification is universally challenging; (2) capabilities across chain steps are dissociable among models, with upstream advantages not consistently transferring downstream; (3) emotion prediction performance does not fully align with sensitivity to appraisal interventions, suggesting some models rely on associative rather than appraisal-based reasoning; and (4) current models have not internalized the mechanisms needed to capture subjective heterogeneity. The main contributions of this work are as follows: • Dataset. We introduce CAREBench, a new benchmark for evaluating LLMs’ emotion understanding, supplying data absent in existing datasets on explicit appraisal reasoning, multi-emotion co-occurrence, and dual-perspective emotion understanding. • Evaluation framework. We propose a process-level evaluation framework covering the full situation \rightarrow reasoning \rightarrow appraisal \rightarrow emotion inferential chain, moving beyond the single-step, single-perspective evaluation paradigms in existing work. This framework exposes the blind spots of relying solely on downstream emotion prediction performance to assess LLMs’ emotion understanding, providing methodological support for more reliable evaluation of LLMs’ affective cognitive capabilities. • Empirical findings. Through systematic evaluation of six LLMs, we provide an empirical basis for comprehensively diagnosing current LLMs’ emotion understanding capabilities, revealing the insufficiency of existing evaluation paradigms in distinguishing surface-level prediction performance from genuine appraisal understanding, with direct implications for building more cognitively interpretable affective models in the future. 2 The CAREBench Dataset 2.1 Data Collection Process models of appraisal theory characterize emotion generation as a multi-stage sequential process, spanning a complete cognitive chain from situation construal through appraisal to emotional response 19 ; 13 . Grounded in this theoretical perspective, we collected annotations covering an inferential chain s\rightarrow r\rightarrow\mathbf{a}\rightarrow e , where s denotes the situational narrative, r the open-ended appraisal reasoning text, \mathbf{a} the appraisal rating vector, e the emotion annotation. In collaboration with psychology experts, we designed a two-phase data collection protocol, with participants recruited via the Prolific platform. In the first phase, participants provided first-person emotional narratives and completed self-annotation; in the second phase, independent third-party annotators read these narratives and completed annotation from an observer’s perspective. Phase 1: First-Person Narratives and Annotation In Phase 1, each participant was asked to recall and describe a recent emotionally significant event. To ensure richness of content while reducing participant burden, we adopted the micro-narratives framework 22 ,

Read the complete text