Experimenting with Affective Computing Models in Video Interviews with Spanish-speaking Older Adults

Josep López Camuñas, Cristina Bustos, Yanjun Zhu, Raquel Ros, Agata Lapedriza · 2025

Read Experimenting with Affective Computing Models in Video Interviews with Spanish-speaking Older Adults by Josep López Camuñas, Cristina Bustos, Yanjun Zhu, Raquel Ros, Agata Lapedriza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16870.txt.

Introduction

López Camuñas, Bustos, Zhu and colleagues tested how well emotion-recognition models work on older adults in 2025. They used video interviews with Spanish-speaking seniors talking to a person or a virtual avatar, and added a new dataset of such interviews.

They compared model outputs for facial expression, text sentiment and smile detection against human labels. Agreement between the models and human annotators was limited, the different channels often disagreed with each other, and individuals varied a lot. The team argues that emotion tools built on younger, uniform groups will misread seniors. Future systems must account for personal and cultural differences if virtual assistants are to support older users well.

Catalog record

CREATORJosep López Camuñas, Cristina Bustos, Yanjun Zhu, Raquel Ros, Agata Lapedriza
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-16870

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Experimenting with Affective Computing Models in Video Interviews with Spanish-speaking Older Adults Josep López Camuñas, Cristina Bustos, Yanjun Zhu, Raquel Ros, Agata Lapedriza Abstract Understanding emotional signals in older adults is crucial for designing virtual assistants that support their well-being. However, existing affective computing models often face significant limitations: (1) limited availability of datasets representing older adults, especially in non-English-speaking populations, and (2) poor generalization of models trained on younger or homogeneous demographics. To address these gaps, this study evaluates state-of-the-art affective computing models—including facial expression recognition, text sentiment analysis, and smile detection—using videos of older adults interacting with either a person or a virtual avatar. As part of this effort, we introduce a novel dataset featuring Spanish-speaking older adults engaged in human-to-human video interviews. Through three comprehensive analyses, we investigate (1) the alignment between human-annotated labels and automatic model outputs, (2) the relationships between model outputs across different modalities, and (3) individual variations in emotional signals. Using both the Wizard of Oz (WoZ) dataset and our newly collected dataset, we uncover limited agreement between human annotations and model predictions, weak consistency across modalities, and significant variability among individuals. These findings highlight the shortcomings of generalized emotion perception models and emphasize the need of incorporating personal variability and cultural nuances into future systems. 1 Introduction Figure 1 : Overview of the analysis pipeline. Video and human annotations are processed through state-of-the-art machine learning models for text sentiment classification, smile detection and facial expression recognition. The outputs are analyzed in different studies to uncover key patterns and insights across multiple signals There is growing interest in developing virtual assistants and social assistive robots for older adults, as these technologies can encourage social interaction and promote well-being [ 3 , 4 , 2 , 6 , 15 , 26 ] . For these systems to effectively engage older adults, they must be capable of perceiving emotional signals and personalizing interactions to accommodate the unique needs of this demographic [ 27 , 34 ] . Despite advancements in emotion perception technologies for human-robot interaction [ 20 , 32 ] , there remains a significant gap in research focused on older adults. This demographic is severely underrepresented in datasets used to train and evaluate affective computing models [ 36 ] , particularly older adults from non-English-speaking populations. As a result, current models often fail to generalize to elderly, leaving their emotional signals poorly understood or inaccurately interpreted. Addressing this gap is crucial, as older adults exhibit unique emotional and behavioral patterns influenced by age-related changes and cultural differences. In this study, we investigate the performance of state-of-the-art facial emotion, text sentiment and smile recognition systems when applied to older Spanish-speaking adults, a group underrepresented in both affective computing research and publicly available datasets. To this end, we analyze emotional signals using two datasets: the Wizard of Oz (WoZ) dataset [ 15 ] , which contains videos of older adults interacting with a virtual avatar, and a novel dataset we collected featuring human-to-human interviews with the same demographic. By introducing this new dataset, we provide a more diverse data distribution to enable broader and more generalizable insights into the emotional expressions of older Spanish-speaking adults. Our work focuses on three key areas of analysis: the alignment between human-annotated labels and automatic model outputs, the relationship between facial emotion signals and speech sentiment, and the impact of individual differences in facial movements and sentiment expression. For this purpose, we leverage state-of-the-art, off-the-shelf models for facial expression recognition, smile detection, and text sentiment classification. An overview of our methodology is presented in Fig. 1 , with a detailed description provided in Sec. 4 . Our findings reveal several challenges in applying generalized emotion perception models to this demographic. Specifically, we observed limited agreement between human annotations and model predictions and significant variability across individuals in facial movements and sentiment expression. These results highlight the limitations of current affective computing models in capturing the nuanced emotional signals of older Spanish-speaking adults and underscore the importance of incorporating personal and cultural variability in model design. By addressing these critical gaps, our work provides new insights into the development of inclusive affective computing systems. Upon acceptance, we will release all code to support further research and advancements in emotion perception for underrepresented populations. 1.1 Small Data Statement This work focuses on analyzing the performance and output alignment of state-of-the-art affective computing models when applied to older adults, with a particular emphasis on Spanish-speaking populations. These populations are notably underrepresented in the development of machine learning models, which has resulted in systems that often fail to capture their unique emotional and behavioral patterns effectively. Studying this group is important because older adults have distinct needs and behaviors that differ significantly from younger or general adult populations. Factors such as age-related changes in facial expressions, speech patterns, and emotional dynamics necessitate tailored approaches for accurate emotion perception. Furthermore, Spanish-speaking older adults represent a significant portion of the global population, yet their cultural and linguistic nuances are seldom reflected in current datasets or models, which tend to focus on English-speaking demographics. Ensuring that affective computing models work well across all demographic groups, including underrepresented groups like the one studied in this paper, is critical for equitable technological advancements. If these systems are to support human well-being, they must accurately understand and respond to the emotional signals of diverse users. Failing to address these gaps risks perpetuating bias and excluding vulnerable populations from the benefits of such technologies. By focusing on this underrepresented group, the study contributes to developing more inclusive, robust, and effective affective computing systems that cater to the diverse needs of a global user base. The goal of this work is to evaluate the performance of state-of-the-art affective computing models on Spanish-speaking older adults, an underrepresented demographic in existing datasets. To address this gap, we relied on the WoZ dataset and, complemented it by collecting a novel dataset to enable broader and more diverse analysis. Our findings reveal limited alignment between human-annotated emotional labels and model outputs, weak agreement between outputs from different modalities, as well as significant variability in facial movements and text sentiment expression across individuals. These results highlight the limitations of generalized models in capturing age- and culture-specific nuances, underscoring the need for inclusive approaches that account for such variability. By identifying these shortcomings, this study contributes to developing robust and equitable emotion perception systems that better serve diverse populations. 2 Related Work Interactive technologies, such as virtual agents and social robots, have demonstrated significant benefits in promoting older adults’ health and social connection [ 40 ] . These technologies feature non-intrusive, emotionally expressive virtual coaches or social robots that encourage users to adopt healthier lifestyles by improving nutritional habits, promoting physical activity, and facilitating social interactions. By fostering these behaviors, they help seniors reduce the risk of chronic diseases, enabling independent and fulfilling lives while also supporting their caregivers [ 6 ] . Augmented and virtual reality technologies have also been explored to enrich older adults’ experiences, including enhancing digital learning [ 13 ] , improving long-term health interventions [ 17 ] , and supporting overall well-being [ 38 ] . Social robots, in particular, have shown

Read the complete text