AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence
Kate M. Lubrano, Faisal Sayed, Akshansh Akshansh, Ankita Rathod, Craver Corbyn Thomas-Smith, Mark E. Whiting, Karina Nguyen, [4pt] Pareto, California, US · 2026
Read AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence by Kate M. Lubrano, Faisal Sayed, Akshansh Akshansh, Ankita Rathod, Craver Corbyn Thomas-Smith, Mark E. Whiting, Karina Nguyen, [4pt] Pareto, California, US (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21739.txt.
Introduction
Lubrano and colleagues introduce AttuneBench, a benchmark for emotional intelligence in language models. It rests on two hundred genuine multi-turn conversations, with turn-by-turn notes on feelings, model conduct, and wanted replies. Across eleven models, rankings on emotion recognition, behavior classification, preference prediction, and response quality prove largely independent. Emotional skill thus splits apart. Predicting the reply a user wants in context tells models apart far more sharply than labeling emotions correctly.
Catalog record
| CREATOR | Kate M. Lubrano, Faisal Sayed, Akshansh Akshansh, Ankita Rathod, Craver Corbyn Thomas-Smith, Mark E. Whiting, Karina Nguyen, [4pt] Pareto, California, US |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106990 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.21739
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence Kate M. Lubrano, Faisal Sayed, Akshansh Akshansh, Ankita Rathod, Craver Corbyn Thomas-Smith, Mark E. Whiting, Karina Nguyen, [4pt] Pareto, California, US Abstract Emotional intelligence (EI), the ability to perceive, understand, and respond appropriately to others’ emotional states, is central to human communication, and increasingly important to assess as LLMs assume conversational roles in everyday life. Existing EI benchmarks rely on synthetic prompts, single-turn cases, or third-party annotation. These approaches do not directly measure how models infer and respond to a participant’s emotional state over the course of a real conversation. We introduce AttuneBench , a benchmark grounded in 200 genuine multi-turn human-model conversations in which participants conversed with anonymized LLMs and provided turn-by-turn annotations of their emotional state, the model’s behavior, and their preferred responses. Across 11 evaluated models, we find that model rankings on emotion recognition, behavioral classification, preference prediction, and judged response quality are largely independent, indicating that emotionally intelligent behavior decomposes into separable capabilities. Preference alignment and response-quality judgments are substantially more model-discriminating than emotion-label accuracy. These results indicate that emotionally intelligent behavior requires predicting what kind of response a specific user wants in context, a distinction that aggregate scoring can obscure and that single-turn or synthetic formats cannot directly capture across turns. AttuneBench provides a framework for assessing each of these capabilities and for diagnosing model-specific strengths and failure modes in emotionally salient conversation. AttuneBench: EI Benchmark Keywords: emotional intelligence, LLM evaluation, conversational benchmark, human-centered AI, empathy modeling 1 Introduction Benchmarks commonly used to assess progress toward general intelligence (e.g., MMLU ( Hendrycks et al. 2021 ) ) largely neglect dimensions of social perception ( Kosinski 2023 ) , emotion’s high-dimensional structure ( Cowen et al. 2019 ) , and contextually appropriate interpersonal behavior ( Rashkin et al. 2019 ) , creating a gap where systems that perform well on traditional metrics may still fail in settings that require emotional awareness or empathy ( Fazzi et al. 2025 ) . Users are increasingly turning to LLMs for advice, emotional support, and personal reflection ( Anthropic 2026b ; Bird et al. 2024 ) . In these contexts, success is less about factual correctness than about responding to emotional states with nuance. Recent survey evidence indicates that approximately 13% of US adolescents and young adults have used LLMs for mental health advice, with the figure rising to 22% among those aged 18–21 ( McBain et al. 2025 ) . Growing research highlights substantial limitations, including inconsistent responses, hallucinated advice, and failure to meet established psychotherapy ethics standards. In high-risk contexts like suicidal ideation, models have demonstrated unreliable alignment that can lead to harmful responses and escalating behavior ( Moore et al. 2026 ) . Yet the benchmarks we use to track LLM progress in this space rely on synthetic vignettes and retrospective annotation, not the real conversations these systems actually have. We introduce AttuneBench , a benchmark for evaluating emotional intelligence (EI) in LLMs through observable conversational behavior: recognizing emotional cues, tracking mood shifts across turns, calibrating tone and guidance, and adapting to user feedback. Because emotional interaction unfolds over time, AttuneBench is built from real multi-turn conversations with human participants. Our contributions: • We release the first open EI benchmark based on real multi-turn human–model conversations, with 200 conversations annotated for PANAS mood trajectories, observed-vs-preferred judgments, pairwise comparisons, and human-edited references. • Across 11 LLMs, we show that EI decomposes into weakly coupled skills: emotion tracking, behavioral judgment, response preference, and attuned generation. • We identify interaction-level failures, including drift across turns, sensitivity to topic and participant profile, and gaps between recognizing user needs and responding appropriately. 2 Related Work Prior work has evaluated emotional and social capabilities in LLMs across emotion recognition, empathy, and theory of mind ( Wang et al. 2023 ; Kosinski 2023 ; Sorin et al. 2024 ; Kusal et al. 2024 ; Chen et al. 2024 ) . However, existing benchmarks typically rely on synthetic data, single-turn formats, model-generated labels, or isolated psychometric probes rather than human-grounded conversational behavior. EQ-Bench ( Paech 2023 ) uses synthetic, single-prompt dialogues and correlates strongly with general reasoning benchmarks, making it unclear whether it isolates EI. EmpatheticDialogues ( Rashkin et al. 2019 ) uses crowdsourced conversations with assigned emotion labels, but evaluates retrieval or single responses rather than turn-level preferences. EmoBench ( Sabour et al. 2024 ) and psychometric adaptations ( Zhang 2024 ; Huang et al. 2024 ) test scenario reasoning or isolated traits, not how models respond across real emotional trajectories. AttuneBench differs by combining genuine multi-turn conversations, participant-grounded annotation, continuous emotional-state measurement, and unified ground truth across evaluation dimensions. 3 Experiment Design 1. Data Collection HP \leftrightarrow Original Model multi-turn dialogue ( n=200 conversations) 2. HP Annotation Pre/Post PANAS, mood-shift tags, binary, pairwise, Q1–Q3, 4-Branch 3. EM Evaluation Evaluated Model predicts HP annotations \to per-metric + Composite Figure 1 : AttuneBench pipeline. HPs converse with an Original Model and annotate each turn (Phases 1–2); Evaluated Models are scored against the HP-grounded annotations (Phase 3). AttuneBench is designed around three core questions: (1) how accurately can LLMs infer a user’s emotional state across a multi-turn conversation; (2) how well do LLMs select or produce responses that align with human preferences in emotional contexts; and (3) are these capabilities independent, or does strong performance on one predict strong performance on others? 3.1 Interaction Structure The benchmark is built around multi-turn conversations. Human Participants (HPs) engage directly with an LLM during data collection; a secondary Evaluated Model (EM) is then scored on how well it reads emotional subtext and how closely its behavioral assessments align with HP judgments. Conversations sample from a pool of Original Models (OMs) to limit single-model bias (Table 8 ). Though the per-OM distribution is unbalanced, Pairwise Accuracy stratified by OM confirms headline rankings are stable across OMs (Appendix G.1 ). 3.2 Task Formulation Topic Selection. Conversation topics were assigned from a curated list containing 50 topics. Our topic taxonomy was informed by prior work on conversational behavior and emotional elicitation. The Conversations We Seek to Avoid ( Sun and Slepian 2020 ) guided the inclusion of sensitive domains (e.g., Money, Politics), while studies of everyday conversational content ( Sehulster 2006 ; Fried-Oken et al. 2015 ) informed the selection of more casual topics (e.g., Friends, Hobbies). Emotional range emerged naturally from randomly assigned topics, individual preferences, and participant mood at session time, rather than through elicitation methods. Participants reported their attitudes toward all topics in advance, confirming coverage across positive and negative reactions. Measurement of Emotional State. To capture pre- and post-conversation emotional state, we adopt the Positive and Negative Affect Schedule (PANAS), a widely validated psychological instrument consisting of 20 affective descriptors rated on a Likert scale ( Watson et al. 1988 ) . PANAS was selected for its empirical grounding, dual-valence structure, and suitability for quantitative analysis. Because PANAS items contain some overlap (e.g., jittery and nervous sit close in valence-arousal space), exact-label scoring would undervalue near-correct emotion predictions. Emotion prediction metrics incorporate partial credit based on continuous affective proximity (Section 4.4 ). Annotation Metric Selection. To ensure theoretical grounding, the benchmark design is informed by the Mayer-Salovey-Caruso Four Branch Model