Saliency-guided Emotion Modeling: Predicting Viewer Reactions from Video Stimuli

Akhila Yaragoppa, Siddharth · 2025

Read Saliency-guided Emotion Modeling: Predicting Viewer Reactions from Video Stimuli by Akhila Yaragoppa, Siddharth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19178.txt.

Introduction

Akhila Yaragoppa and Siddharth published a research paper in 2025 on predicting how viewers will feel about a video from where their eyes are drawn. Emotion tech usually leans on self-reports, faces, or body sensors. Their approach instead measures the video's salient areas, the spots that grab attention, using a deep saliency model plus face-action software.

Two simple features did the work: how much of the frame is salient, and how many salient spots compete. Videos with many spots tended to spark pleasant, calm feelings. One dominant spot leaned toward unpleasant, charged feelings. Self-reports often clashed with face-based readings, a caution for the field.

Catalog record

CREATORAkhila Yaragoppa, Siddharth
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19178

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Saliency-guided Emotion Modeling: Predicting Viewer Reactions from Video Stimuli Akhila Yaragoppa, Siddharth Abstract Understanding the emotional impact of videos is crucial for applications in content creation, advertising, and Human-Computer Interaction (HCI). Traditional affective computing methods rely on self-reported emotions, facial expression analysis, and biosensing data, yet they often overlook the role of visual saliency—the naturally attention-grabbing regions within a video. In this study, we utilize deep learning to introduce a novel saliency-based approach to emotion prediction by extracting two key features: saliency area and number of salient regions. Using the HD2S saliency model and OpenFace facial action unit analysis, we examine the relationship between video saliency and viewer emotions. Our findings reveal three key insights: (1) Videos with multiple salient regions tend to elicit high-valence, low-arousal emotions, (2) Videos with a single dominant salient region are more likely to induce low-valence, high-arousal responses, and (3) Self-reported emotions often misalign with facial expression-based emotion detection, suggesting limitations in subjective reporting. By leveraging saliency-driven insights, this work provides a computationally efficient and interpretable alternative for emotion modeling, with implications for content creation, personalized media experiences, and affective computing research. 1 Introduction Understanding the emotional impact of videos and films on viewers has long been a subject of significant research interest due to its wide-ranging applications, including in advertising, video retrieval, and content summarization [ 1 ] . Traditionally, researchers have relied on participant self-reported scores, facial expression analysis, electroencephalogram (EEG) recordings, and other physiological measurements to study the emotions elicited by watching videos. However, these approaches suffer from subjectivity, computational complexity, and a lack of focus on key visual elements. One critical gap in affective computing research is the role of visual saliency—the naturally attention-grabbing regions in a video that may significantly influence emotional responses. This study utilized a deep learning model to detect salient regions in a video and introduces two novel interpretable features—“saliency area” and “number of salient regions”—derived from saliency maps of video content. Unlike traditional methods, these saliency-based features offer a new perspective on analyzing and predicting the emotional responses elicited by videos. To our knowledge, this is the first attempt to investigate the potential of using saliency features to correlate with viewer emotions and derive actionable insights for content creation. The biggest advantage of using saliency-based features over other methods that have previously been explored is that we would only need to look at the salient regions of the video to understand the emotion elicited. This would not only decrease the computational processing but will also aid content creators in figuring out what kind of salient features in the videos may induce which emotions in the users. To achieve this, we also utilize facial expression analysis and establish relationships between the saliency-based video features and users’ reported emotions. This research work makes the following key contributions: 1. Video frames containing multiple salient regions are more likely to evoke emotions characterized by high valence and low arousal. 2. Videos that evoke emotions with low valence and high arousal often focus on a single salient region at a time. 3. Self-reports often misalign with facial expression-based emotions, suggesting limitations in subjective reporting. 2 Related Works 2.1 Video Saliency Prediction Saliency prediction is concerned with identifying the elements within a scene that naturally draw human attention. There are two main types of models for saliency prediction: saliency prediction models that estimate where observers will focus their gaze [ 2 ] , and salient object detection models that identify objects of interest against a background [ 3 ] . These models can further be classified into static saliency for images and dynamic saliency for videos. Static saliency models have evolved from early hand-crafted features [ 4 ] to more advanced CNN-based models that integrate deep learning techniques [ 5 , 6 , 7 ] leading to better performance. The introduction of large datasets further improved their accuracy. Dynamic saliency models for videos address the additional challenge of capturing temporal changes. Early approaches adapted static models to video by analyzing frames independently, but these were soon outperformed by models designed to process spatial and temporal data simultaneously. Some of the leading models for dynamic saliency prediction are [ 8 , 9 , 10 ] . We use the HD^{2}S model proposed in [ 10 ] , a domain-agnostic architecture adaptable to diverse stimuli. Its key strength lies in generalizing across datasets without fine-tuning, enabled by gradient reversal layers that promote domain-independent feature learning. HD^{2}S outperforms state-of-the-art methods on three of five metrics and ranks second-best on the remaining two in the DHF1K benchmark [ 11 ] . However, it may struggle with small objects or subtle motion and demands substantial computational resources at higher resolutions. The model is 116 MB in size with a runtime of 0.027 seconds. Further architectural and parameter details are available in [ 10 ] . 2.2 Facial Action Units Extraction Numerous methods exist for automatically identifying facial action units (AUs) from facial expressions based on the Facial Action Coding System (FACS) [ 12 ] , using a range of machine learning and computer vision techniques to interpret facial behavior. Traditional approaches relied on handcrafted features to capture facial muscle movements [ 13 ] , while recent deep learning methods leverage convolutional neural networks (CNNs) to learn features from large annotated datasets, improving the accuracy and robustness of AU detection [ 14 , 15 , 16 ] . We use the OpenFace library [ 17 ] , a widely adopted open-source toolbox for facial behavior analysis, offering tools for facial landmark detection, head pose estimation, gaze tracking, and AU detection. Its AU detection module, based on [ 18 ] , combines geometric and appearance-based features from video sequences to train machine learning models capable of real-time AU recognition. Among the many available methods, we selected [ 18 ] for its strong performance metrics and ease of integration. 3 Methodology Figure 1 : Distribution of participants’ self-reported emotions in the MAHNOB-HCI Dataset on the Emotion Circumplex Model 3.1 Dataset Dataset and trials : Several datasets have been gathered and publicly released to explore the link between elicitation videos and the emotions they provoke, providing data on participants’ physiological responses, facial video recordings, and the specific videos they watched during the study. We choose the MAHNOB-HCI dataset [ 19 ] to conduct our analysis due to its wide adoption in the affective computing community. This dataset was collected from 30 participants watching 20 videos each. Excluding the trials with missing information, we finally use 527 trials, for which participants reported emotional responses (after each trial) and facial videos are available. We use the facial videos, felt valence score, felt arousal score, and the corresponding elicitation videos for our study. For each trial, we utilize the facial video, the corresponding valence and arousal labels reported by the participant, and the emotion elicitation video that the participant watched during the trial. Facial & elicitation video processing : We sample the frames in the video at a frequency of 2 frames per second, as the change in expressions in this duration is insignificant. Participant labels : The valence and arousal labels reported by the participants are in the range 1-9 according to the

Read the complete text