EmoNet-Face : An Expert-Annotated Benchmark for Synthetic Emotion Recognition
Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Krishna Kalyan, Kourosh Nadi, Huu Nguyen, Kristian Kersting, Sören Auer · 2025
Read EmoNet-Face : An Expert-Annotated Benchmark for Synthetic Emotion Recognition by Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Krishna Kalyan, Kourosh Nadi, Huu Nguyen, Kristian Kersting, Sören Auer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20033.txt.
Introduction
Christoph Schuhmann and nine co-authors built a benchmark for how well AI reads emotions from faces, in a 2025 paper. Good human-AI interaction needs accurate reading of feelings. Current test sets cover few emotions and blur close ones, such as shame and embarrassment. Many also use messy images and skewed samples.
Their suite, EmoNet-Face, starts with a forty-category map of emotions drawn from classic research. Three large data sets use AI-generated faces with full, clear expressions. Samples are balanced across ethnicity, age, and gender by design. Expert annotators labeled the images in depth. The team also built a model that matched human expert scores on the benchmark. The full suite is public, offering a base for fairer emotion tools.
Catalog record
| CREATOR | Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Krishna Kalyan, Kourosh Nadi, Huu Nguyen, Kristian Kersting, Sören Auer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20033
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EmoNet-Face : An Expert-Annotated Benchmark for Synthetic Emotion Recognition Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Krishna Kalyan, Kourosh Nadi, Huu Nguyen, Kristian Kersting, Sören Auer Abstract Effective human-AI interaction relies on AI’s ability to accurately perceive and interpret human emotions. Current benchmarks for vision and vision-language models are severely limited, offering a narrow emotional spectrum that overlooks nuanced states (e.g., bitterness, intoxication) and fails to distinguish subtle differences between related feelings (e.g., shame vs. embarrassment). Existing datasets also often use uncontrolled imagery with occluded faces and lack demographic diversity, risking significant bias. To address these critical gaps, we introduce EmoNet-Face , a comprehensive benchmark suite. EmoNet-Face features: (1) A novel 40-category emotion taxonomy, meticulously derived from foundational research to capture finer details of human emotional experiences. (2) Three large-scale, AI-generated datasets ( EmoNet-Face HQ , EmoNet-Face Binary , and EmoNet-Face Big ) with explicit, full-face expressions and controlled demographic balance across ethnicity, age, and gender. (3) Rigorous, multi-expert annotations for training and high-fidelity evaluation. (4) We build EmpathicInsight-Face , a model achieving human-expert-level performance on our benchmark. The publicly released EmoNet-Face suite—taxonomy, datasets, and model—provides a robust foundation for developing and evaluating AI systems with a deeper understanding of human emotions. 1 Introduction The advent of highly capable Large Language Models (LLMs) such as ChatGPT [ 34 ] has transformed human-computer interaction, a transformation further accelerated by advances in voice synthesis and multimodal reasoning [ 35 ] . Modern voice interfaces now enable emotionally expressive AI conversations [ 44 , 25 ] , leading to heightened expectations for AI systems with genuine emotional intelligence. As AI platforms like Replika [ 29 ] , Character.ai [ 9 ] , and Kajiwoto [ 22 ] are increasingly used for companionship and support [ 10 ] , and as LLMs are consulted for mental health advice [ 47 , 23 , 16 , 46 ] , it becomes ever more critical for these systems to accurately interpret nonverbal emotional cues, especially facial expressions. Misreading users’ emotions can erode trust and have negative psychological repercussions [ 8 ] . At the same time, as AI systems become more embodied—through avatars and realistically synthesized faces—another crucial challenge emerges: enabling humans to correctly perceive and interpret the emotional states of these digital agents. For emotionally rich and affective interaction, the expressions generated by machine avatars must be legible and interpretable to human users, ensuring users understand not only the intentionality but also the emotional stance and nuance behind an AI’s responses. This reciprocity is critical for building trust, rapport, and effective communication in human-AI partnerships. Truly emotionally aware AI thus requires a bidirectional capability: machines must recognize not only basic emotions [ 13 ] but also complex and subtle affective states [ 36 , 12 , 39 ] , while synthetic emotions generated by AI avatars must be expressively rich and clearly understood by human counterparts. However, current benchmarks are limited in size, their spectrum of emotions, the diversity of groups included, and the quality of their annotations, constraining progress toward AI systems capable of authentic, context-sensitive empathy and expressiveness [ 11 ] . To address these gaps, we introduce EmoNet-Face , an expert-annotated data suite for fine-grained facial emotion recognition grounded in a novel, expansive 40-category emotion taxonomy. This taxonomy, developed by mining the "Handbook of Emotions" [ 27 ] and refined with psychological expertise, captures a broader and more detailed array of human emotional states than previous efforts. Using state-of-the-art text-to-image models [ 6 , 32 ] , we generated a series of structured, demographically balanced, high-quality synthetic datasets, covering a wide range of age, ethnicity, and gender. This includes a pretraining set of over 203,000 images ( EmoNet-Face Big ), a fine-tuning dataset ( EmoNet-Face Binary ) with almost 20k images and more than 62,000 human expert binary emotion annotations, and an evaluation benchmark ( EmoNet-Face HQ ) of 2,500 images meticulously rated by psychology experts using continuous scales across all 40 emotion categories. Alongside these datasets, we present EmpathicInsight-Face , two models trained on our suite that achieve human-expert-level performance on the EmoNet-Face HQ benchmark and outperform concurrent models. Our main contributions are as follows: (i) We introduce a comprehensive 40-category emotion taxonomy, refined through expert consultation, to capture fine-grained human emotional states; (ii) We construct diverse synthetic expert-annotated image datasets—spanning pretraining, fine-tuning, and benchmarking test sets—using state-of-the-art text-to-image models; (iii) We develop EmpathicInsight-Face models that match human performance on fine-grained emotion recognition; (iv) We openly release our datasets and models to foster research on emotions in AI. 1 1 1 We openly release our data and code at our HuggingFace and GitHub repositories . Table 1 : Comparison of facial emotion recognition datasets, including synthetic generation and expert annotation. Open license means CC-BY 4.0 or similar. Dataset Name Open License Size Emotions Synthetic Expert Labels Continuous Diversity FERG-DB [ 2 ] ✗ 0 55,767 0 7 ✗ ✗ ✗ ✗ FERG-3D-DB [ 1 ] ✗ 0 39,574 0 7 ✗ ✗ ✗ ✗ SZU-EmoDage [ 19 ] ✓ 000 840 0 8 ✓ ( ✓ ) ✗ ✗ EmotionNet [ 17 ] ✗ 00, N/A 0 6 ✗ ✗ ✗ N/A EMOTIC [ 24 ] ✗ 0 18,313 26 ✗ ✗ ( ✓ ) ( ✓ ) FindingEmo [ 50 ] ✓ 0 25,869 0 8 ✗ ✗ ✓ ( ✓ ) AffectNet [ 33 ] ✗ 450,000 0 8 ✗ ✗ ✗ ( ✓ ) ours EmoNet-Face Big ✓ 203,201 40 ✓ ✗ ✗ ✓ EmoNet-Face Binary ✓ 0 19,999 40 ✓ ✓ ✗ ✓ EmoNet-Face HQ ✓ 00 2,500 40 ✓ ✓ ✓ ✓ 2 Related Work Understanding human emotion remains a complex task, grounded in diverse psychological theories and demanding robust data for computational modeling. Emotions are challenging as they are not universal—they are socially constructed, shaped by cultural norms and context (further elaborated in [ 3 , 4 ] and App. A.1 ). This section outlines key theoretical frameworks, examines existing emotion taxonomies and datasets, and identifies the specific gaps that EmoNet-Face addresses. Existing Emotion Taxonomies. Theories of emotion span a spectrum from universal, biologically rooted models to constructivist perspectives, with the latter emphasizing that emotions are shaped by context and culture rather than being innate, static programs [ 3 , 4 ] . This conceptual diversity has direct consequences for how emotions are categorized and labeled in practice. Traditional taxonomies provide important foundations but often lack granularity. Ekman’s model expanded to include emotions like amusement and shame [ 14 ] , but remains limited to discrete, universal states. Parrott’s hierarchical structure [ 38 ] and Plutchik’s Wheel [ 40 ] offer more coverage but overlook distinctions critical for real-world applications. For example, Plutchik’s dyadic model emphasizes emotional opposites and blends but may oversimplify ambiguous affective states. Other approaches include Panksepp’s neurobiologically rooted systems (e.g., SEEKING, FEAR, LUST) [ 37 ] , and Izard’s emphasis on distinct expressive and neural signatures for emotions like joy, guilt, and interest [ 20 , 21 ] . Recent efforts have enriched positive emotion taxonomies—e.g., Shiota et al.’s physiology-based framework [ 45 ] and Weidman and Tracy’s focus on subjective feeling states [ 48 ] . Despite