GEMS : G roup E motion profiling through M ultimodal S ituational understanding
2025
Read GEMS : G roup E motion profiling through M ultimodal S ituational understanding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22393.txt.
Introduction
Anubhav Kataria and co-authors introduce GEMS, a framework for profiling group emotion from multimodal scene understanding, in a 2025 paper. Reading a social scene means judging not just each face, but the mood of the group and the event as a whole. Most earlier benchmarks stuck to simple interactions and group-level labels over time.
GEMS uses a multimodal swin-transformer design with a special attention module, taking in the scene, group members, and context together. It predicts discrete and continuous emotions, including valence and arousal, at individual, group, and event levels. The team also extends an existing dataset with finer labels, called VGAF-GEMS. Benchmarks show the framework beating adapted leading models, pointing toward richer emotion analysis for group settings.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22393
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GEMS : G roup E motion profiling through M ultimodal S ituational understanding Abstract Understanding individual, group and event level emotions along with contextual information is crucial for analyzing a multi-person social situation. To achieve this, we frame emotion comprehension as the task of predicting fine-grained individual emotion to coarse grained group and event level emotion. We introduce GEMS that leverages a multimodal swin-transformer and S3Attention based architecture, which processes an input scene, group members, and context information to generate joint predictions. Existing multi-person emotion related benchmarks mainly focus on atomic interactions primarily based on emotion perception over time and group level. To this end, we extend and propose VGAF-GEMS to provide more fine grained and holistic analysis on top of existing group level annotation of VGAF dataset [ 1 ] . GEMS aims to predict basic discrete and continuous emotions (including valence and arousal) as well as individual, group and event level perceived emotions. Our benchmarking effort links individual, group and situational emotional responses holistically. The quantitative and qualitative comparisons with adapted state-of-the-art models demonstrate the effectiveness of GEMS framework on VGAF-GEMS benchmarking. We believe that it will pave the way of further research. The code and data is available at: https://github.com/katariaak579/GEMS Index Terms — Social Interaction, Group Emotion. 1 Introduction Comprehensive situational interpretation requires emotional understanding over time as human emotion is associated with multiple entities (e.g. persons, emotion, actions, interaction and scenes) interacting dynamically over time. Although progress has been made in several benchmarking efforts for individual level [ 2 ] , context level [ 3 ] , group level [ 4 ] , event level [ 5 ] emotion understanding, however, current state-of-the-art models yet struggle to discriminate between multimodal as well as multiple entities (e.g. persons, actions, and scenes) based emotion understanding such as in a birthday situation where the situation level emotion is happy/excited. However, each individual may not be expressing happy expression throughout the whole situation. Group-level emotional understanding has a wide range of potential applications, including providing automatic feedback for lectures [ 6 ] , monitoring student engagement in classrooms [ 7 ] , event detection [ 8 ] , group cohesion [ 9 , 10 ] , surveillance, ranking images from events [ 11 ] , and summarizing events [ 11 ] . Fig. 1 : GEMS. A brief overview of GEMS framework. Given any of the defined situations (eg: Celebration-party, Fighting), the framework aims to link participant’s individual emotion, group emotion, situational emotion. Previous work on group-level emotion analysis [ 4 , 12 , 13 ] primarily focused on individual-level analysis followed by aggregation, but lacked annotations capturing temporal dynamics and situational context critical for accurate group-level perception. To this end, we propose VGAF-GEMS, a new benchmark to objectively test different levels of emotion perception and evaluate the compositional ability of individual to group and event level emotion understanding through the lens of time over a situation (See Fig. 1 ). We adopt VGAF [ 1 ] , a group level emotion recognition dataset that consists of coarse grained group level emotion annotations with event level tags. Sourced from youtube clips, VGAF videos present challenging scenarios with frequent shot changes, fast action sequences, multi-event complex situations, role switching between the speaker and listener, all entangled through time. We parse these complex videos with dense individual level emotion annotations capture multiple and changing aspects of the situations in the video along with group level pooling with situational information. We build a rich annotations to create a challenging and high-quality benchmark test suite that requires binding the correct emotion concepts beyond a individual or group level emotion. • We propose G roup E motion profiling through M ultimodal S ituational understanding aka GEMS that covers depth and breadth of emotional profiling in a given situation. • To the best of our knowledge, we are the first to propose VGAF-GEMS , a densely annotated group emotion profiling benchmark that covers the compositionality of individual, situation, contextual aspects of people interacting in a short video. Please note that we annotate images from a third-person perspective following the literature and we incorporate a novel semi-automatic interface-based data annotation strategy using Emolysis [ 14 ] toolkit. • We perform comprehensive analysis and benchmark the proposed dataset utilizing state-of-the-art emotion recognition algorithms. We benchmark the datasets for two different learning paradigms, i.e. , zero-shot and fully supervised. The significant drop in performance (i.e., \sim\textbf{54.80} w.r.t VGAF dataset with supervised framework) directly indicate that our dataset will be a valuable asset to pursue further research in this domain. 2 Background Prior work in group-level emotion analysis can be divided into three main categories: bottom-up , top-down , and hybrid approaches. Bottom-up approaches focus on analyzing individual group members’ emotions and aggregating their contributions to assess the overall group mood [ 15 ] . In contrast, top-down approaches begin with global factors affecting the group and examine how these factors shape the perception of the group’s emotional state [ 16 ] . Hybrid approaches combine both group-level and individual-level information to estimate the group’s emotional state [ 12 , 4 , 17 , 1 ] . A comprehensive description of the prior works is indicated at [ 13 ] . Fig. 2 : Data statistics for the GEMS-VGAF dataset. The figure presents the class-wise distribution of individual emotions, group emotions, and situational contexts in the proposed dataset. Note, Quar: Quarrelling, CasGat: Casual Gathering, GrpAct: Group Activities and CelPar:Celebration Party. The y-axis indicates the number of samples corresponding to each class. An initial work [ 18 ] was proposed back in 2015 on automatic recognition of group emotions using web crawled static images. These images have groups of individuals and were labeled according to six levels of happiness intensity (i.e in arousal axis of emotion). Later works in this domain expanded the research into a multimodal analysis using audio visual inputs along with extending it to valence axis of emotion: i.e. negative, neutral, and positive [ 19 , 20 , 21 , 22 , 23 ] . Related research has examined multi-level emotion analysis from different perspectives. Yilmaz et al. [22] investigated sentiment analysis with multiple emotion labels across 100 languages. Their work shows how multiple emotions can exist simultaneously in a single piece of content - much like how our videos contain multiple individuals each expressing different emotions that together create a complex group emotional state. In another study, Giritlioglu et al. [ 24 ] analyzed video personality traits by looking at both what people do and how they look. This is similar to our work, where we combine individual emotions to understand group emotions using multiple types of information. However, these efforts faced two significant limitations: (1) the works did not consider the temporal dynamics and shifts in group emotions across consecutive video segments. As a result, the emotion annotations were temporally independent, and (2) the papers did not contextualize group emotions within the framework of group interactions. To address these issues, Lei et al. [ 25 ] annotated 2-minute segments of group interactions, incorporating temporal context from interaction videos. Nonetheless, the extended duration of these segments hindered a more nuanced analysis of the dynamics of group emotions.