ℰ m o t i o n ℋ a l l u c e r \mathcal{E}motion\mathcal{H}allucer : Evaluating Emotion Hallucinations in Multimodal Large Language Models
Bohao Xing, Xin Liu, Guoying Zhao, Chengyu Liu, Xiaolan Fu, Heikki Kälviäinen · 2025
Read ℰ m o t i o n ℋ a l l u c e r \mathcal{E}motion\mathcal{H}allucer : Evaluating Emotion Hallucinations in Multimodal Large Language Models by Bohao Xing, Xin Liu, Guoying Zhao, Chengyu Liu, Xiaolan Fu, Heikki Kälviäinen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11405.txt.
Introduction
Bohao Xing and co-authors build the first test set for emotion mistakes in multimodal chatbots in a 2025 paper. Such models can misread feelings or invent them, a flaw called hallucination. No shared yardstick existed for that flaw in the emotion field.
Their benchmark, EmotionHallucer, draws on emotion psychology and real-world scenes with sound and sight. Paired true and fake questions probe each model in a strict yes-no form. Tests of many models found wide gaps, with closed models ahead of open ones. Skill at book facts beat skill at live scenes. A fix they propose lifts scores by a clear margin.
Catalog record
| CREATOR | Bohao Xing, Xin Liu, Guoying Zhao, Chengyu Liu, Xiaolan Fu, Heikki Kälviäinen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11405
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ℰ m o t i o n ℋ a l l u c e r \mathcal{E}motion\mathcal{H}allucer : Evaluating Emotion Hallucinations in Multimodal Large Language Models Bohao Xing, Xin Liu, Guoying Zhao, Chengyu Liu, Xiaolan Fu, Heikki Kälviäinen Abstract Emotion understanding is a critical yet challenging task. Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced their capabilities in this area. However, MLLMs often suffer from “hallucinations”, generating irrelevant or nonsensical content. To the best of our knowledge, despite the importance of this issue, there has been no dedicated effort to evaluate emotion-related hallucinations in MLLMs. In this work, we introduce EmotionHallucer , the first benchmark for detecting and analyzing emotion hallucinations in MLLMs. Unlike humans, whose emotion understanding stems from the interplay of biology and social learning, MLLMs rely solely on data-driven learning and lack innate emotional instincts. Fortunately, emotion psychology provides a solid foundation of knowledge about human emotions. Building on this, we assess emotion hallucinations from two dimensions: emotion psychology knowledge and real-world multimodal perception. To support robust evaluation, we utilize an adversarial binary question–answer (QA) framework, which employs carefully crafted basic and hallucinated pairs to assess the emotion hallucination tendencies of MLLMs. By evaluating 38 LLMs and MLLMs on EmotionHallucer, we reveal that: i) most current models exhibit substantial issues with emotion hallucinations; ii) closed-source models outperform open-source ones in detecting emotion hallucinations, and reasoning capability provides additional advantages; iii) existing models perform better in emotion psychology knowledge than in multimodal emotion perception. As a byproduct, these findings inspire us to propose the PEP-MEK framework, which yields an average improvement of 9.90% in emotion hallucination detection across selected models. Resources will be available at https://github.com/xxtars/EmotionHallucer . Inevitably, emotions are inseparable from the idea of good and evil. Antonio Damasio, The Feeling of What Happens . 1 Introduction Emotion understanding is one of the most fundamental yet challenging tasks in artificial intelligence koelstra2011deap ; hakak2017emotion . Over the past decades, this field has attracted significant attention from the research community nandwani2021review ; li2020deep ; el2011survey ; ezzameli2023emotion ; rahdari2019multimodal ; xing2024emo . Much of the existing work has focused on independent sub-tasks across different modalities: in the text modality, tasks such as sentiment analysis wankhade2022survey and emotion cause detection lee2010text ; in the image modality, facial expression recognition li2020deep and affective scene analysis zhao2021affective ; in the speech modality, speech emotion recognition wani2021comprehensive ; and in the video modality, multimodal emotion recognition abdullah2021multimodal , dynamic facial expression recognition zhao2021former , and body gesture-based emotion recognition liu2021imigue , among others. Recently, MLLMs have demonstrated remarkable capabilities in text and vision understanding alayrac2022flamingo ; li2023blip ; achiam2023gpt , and have begun to play an increasingly important role in emotion understanding zhang2023dialoguellm ; lian2023explainable ; lian2025affectGPT ; cheng2024emotion ; xing2024emo . These models offer the potential for unified, multimodal emotion understanding that transcends traditional unimodal and single-task approaches. However, despite their strong capabilities, MLLMs often generate incorrect or ungrounded responses based on textual or visual inputs li2023evaluating ; tong2024eyes ; petryk2024aloha . This issue is commonly called “hallucination” rohrbach2018object . Hallucination is generally categorized into two types bai2024hallucination : (1) factuality hallucination, where outputs conflict with real-world facts; (2) faithfulness hallucination, where outputs diverge from input instructions, provided context, or exhibit internal inconsistencies. In response to these challenges, growing attention has been paid to analyzing and mitigating hallucinations in MLLMs. Existing hallucination benchmarks, however, are primarily designed for general-purpose tasks wang2024videohallucer , leaving hallucinations in emotion understanding tasks largely unexplored. Unlike general tasks, emotion understanding involves subjective and psychological processes, often shaped by cognitive appraisal, context, and social cues, which makes emotion-related hallucinations harder to detect. On the other hand, human emotion arises from a combination of innate biological mechanisms and lifelong social learning zeidner2003development , which makes it challenging to model, as shown in Figure 1(a) . In contrast, MLLMs rely on data-driven learning, and lack the embodied and experiential grounding that humans use to interpret emotions naturally and intuitively. Fortunately, learning how emotion develops may help us understand more about emotion itself shiota2017emotion . Decades of research in psychology have offered rich insights into how humans perceive, process, and reason about emotions niedenthal2017psychology , offering a valuable knowledge source to support more reliable emotion understanding in MLLMs. Motivated by these observations, we first consider how emotion-related hallucinations should be defined and categorized. Unlike general hallucinations, emotion hallucinations tend to be more complex, as emotion understanding involves not only objective perception but also psychological and sociological reasoning zaki2012neuroscience . Given this complexity, we propose EmotionHallucer , the first benchmark specifically designed to evaluate emotion hallucinations. EmotionHallucer targets two key aspects: hallucinations related to emotion psychology knowledge (focusing on factuality hallucination) and hallucinations in real-world multimodal emotion understanding (focusing on faithfulness hallucination), as illustrated in Figure 1(b) . To ensure reliable evaluation and reduce confounding factors li2023evaluating ; zhang2023language , we adopt a binary QA-based evaluation framework li2023evaluating ; wang2024videohallucer . Specifically, we construct adversarial QA pairs tong2024eyes , where each pair consists of a basic and an intentionally hallucinated question to test models. To mitigate language bias, we balance “yes” and “no” answers, and provide concise explanations to reduce misinterpretation. By evaluating 38 LLMs and MLLMs on EmotionHallucer, our analysis leads to three mian findings: First , most current models exhibit substantial issues with emotion hallucinations. Second , closed-source models outperform open-source ones in detecting emotion hallucinations, and reasoning capability provides additional advantages. Third , existing models perform better in emotion psychology knowledge than in multimodal emotion perception. Building on these findings, we propose PEP-MEK , a plug-and-play framework that incorporates both modality-specific and emotional knowledge to mitigate emotion hallucinations. Experimental results show that applying PEP-MEK leads to a significant performance boost on EmotionHallucer, with an average improvement of 9.90%. We believe this framework can support future research and development in the detection and mitigation of emotion hallucinations in MLLMs. (a) (b) Figure 1 : Emotion understanding differences and the EmotionHallucer. (a) The distinction between how humans and MLLMs understand emotions. Based on the component process model scherer2009dynamic and dynamic systems approach lewis2005bridging , human emotion understanding involves dynamic interactions among cognitive appraisals, physiological changes, feelings, and behaviors. In contrast, MLLMs rely on data-driven learning from external behavioral cues, which limits their ability to accurately infer the underlying emotional states. (b) EmotionHallucer is organized along two main dimensions, Emotion Knowledge and Multimodality Perception, and includes seven subcategories across four modalities. 2 Related Work 2.1 Hallucination in Natural Language Processing Generative models in Natural Language Processing (NLP), particularly Large Language Models (LLMs), have demonstrated impressive performance across a wide range of language generation tasks. However, a major challenge remains: these models may occasionally produce text that is inaccurate, irrelevant, or illogical, a phenomenon commonly referred to as “hallucination” in NLP. Hallucination typically refers to instances where the generated content is nonsensical or deviates from the intended meaning or source material filippova2020controlled . In the NLP community, this issue is empirically categorized into two types bai2024hallucination : 1) Factuality hallucination, which highlights inconsistencies between generated output and verifiable real-world facts, often manifesting as factual errors or