EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language

Phoebe Chua, Cathy Mengying Fang, Takehiko Ohkawa, Raja Kushalnagar, Suranga Nanayakkara, Pattie Maes · 2025

Read EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language by Phoebe Chua, Cathy Mengying Fang, Takehiko Ohkawa, Raja Kushalnagar, Suranga Nanayakkara, Pattie Maes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17090.txt.

Introduction

Phoebe Chua and co-authors release a data set on emotion in sign language in a 2025 paper. In spoken tongues, tone carries feeling, but in sign language the face and hands serve grammar and emotion at once, which makes feeling hard to read. Their EmoSign set labels two hundred American Sign Language videos for sentiment and emotion.

Three Deaf signers with interpreting experience did the labeling and wrote open notes on the cues they saw. Starter models for sorting sentiment and emotion come with the set. Readers in accessibility research get the first benchmark for machine reading of feeling in sign.

Catalog record

CREATORPhoebe Chua, Cathy Mengying Fang, Takehiko Ohkawa, Raja Kushalnagar, Suranga Nanayakkara, Pattie Maes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17090

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language Phoebe Chua, Cathy Mengying Fang, Takehiko Ohkawa, Raja Kushalnagar, Suranga Nanayakkara, Pattie Maes Abstract Unlike spoken languages where the use of prosodic features to convey emotion is well studied, indicators of emotion in sign language remain poorly understood, creating communication barriers in critical settings. Sign languages present unique challenges as facial expressions and hand movements simultaneously serve both grammatical and emotional functions. To address this gap, we introduce EmoSign, the first sign video dataset containing sentiment and emotion labels for 200 American Sign Language (ASL) videos. We also collect open-ended descriptions of emotion cues. Annotations were done by 3 Deaf ASL signers with professional interpretation experience. Alongside the annotations, we include baseline models for sentiment and emotion classification. This dataset not only addresses a critical gap in existing sign language research but also establishes a new benchmark for understanding model capabilities in multimodal emotion recognition for sign languages. The dataset is made available at https://huggingface.co/datasets/catfang/emosign . * * footnotetext: These authors contributed equally to this work. 1 Introduction The emotional content of a speech comes not only from its linguistic content but also how it is spoken—from pitch and intonations to non-verbal expressions [ 12 ] . For sign language, however, emotional indicators are less understood and studied [ 15 , 17 , 30 ] . This ambiguity has practical negative consequences from misinterpretations of signers’ feelings to causing biases and prejudices in legal settings [ 42 ] and emergency departments [ 20 ] . The root of the ambiguity comes from the differences in the makeup of sign languages. Sign languages are comprised of five parameters: Handshape, Place of Articulation (where the sign is made), Movement (how the articulators move), Orientation (the hands’ relation towards the Place of Articulation), and Non-manual behaviors (what the body and face are doing). They function similarly to the cavities, articulators and features of spoken languages [ 45 , 37 ] . The non-manual behaviors serve as linguistic (e.g., grammatical) markers in addition to conveying emotions. One category of such non-manual behaviors is facial expressions. For example, “puffing of one’s face” indicates intensity and “raised eyebrows” indicates a yes/no question [ 45 ] . At the same time, one’s facial expressions also convey emotions, which are applied on top of the linguistic meaning [ 30 ] . Other ways of expression emotions in sign languages include the hand movements, such as the tempo, rhythm, and size of the movements [ 39 ] . For example, the duration of sentences was shorter, the sign movements were more angular, and their movement paths shortened when expressing anger as compared to a neutral condition. Emotion recognition is a common task in the machine learning community. Datasets and models have been constructed to recognize the sentiment and emotions based on text [ 19 , 43 ] , speech [ 33 , 6 ] , facial expressions [ 4 ] , and a combinations of multiple input modalities [ 51 , 7 , 38 ] . Emerging multi-modal foundation models have demonstrated abilities to understand context of the videos [ 31 ] . Few have extended the capabilities of pre-trained models to understand affect via fine-tuning and instruction-tuning [ 24 , 8 ] on new datasets and benchmarks [ 27 ] . Emotion recognition on sign languages, however, remain sparse. Sign language translation itself is already difficult. In addition, current approaches fail to recognize the complex linguistic and affective functions of facial expressions during signing. To address this critical research gap, we introduce EmoSign, the first comprehensive dataset containing sentiment and emotion labels for American Sign Language (ASL) videos. The dataset includes 200 ASL video clips annotated by 3 Deaf ASL signers with professional interpretation experience, who provided: (1) Overall sentiment ratings on a 7-point scale, (2) Presence and intensity ratings for 10 distinct emotion categories (3) Detailed descriptions of specific emotion cues. Unlike existing sign language datasets that focus primarily on translation capabilities, EmoSign specifically targets the affective dimensions of signing. Our work contributes to both sign language accessibility and emotion recognition research by: 1. Providing the first dedicated dataset for studying emotional expression in ASL 2. Documenting descriptions of how emotions manifest through manual and non-manual components through the lens of native signers 3. Establishing baseline model performance for sentiment analysis and emotion classification tasks in sign language The impact of our work extends beyond emotion recognition of sign languages. Studying how emotions can be recognized from a mix of non-manual and manual components allows us to study emotion differently. This also presents a new technical challenge to the AI/ML community with a challenging new benchmark for multimodal AI systems that tests their ability to distinguish between grammatical and affective functions of emotional cues in sign languages. 2 Related Work 2.1 Emotion recognition Datasets for emotion recognition are typically annotated using either discrete or dimensional emotion frameworks. Discrete emotion labels are based on theories of basic emotion, which suggest that there are a limited number of emotion states (e.g., fear, happiness) associated with distinct expressions and physiological states [ 14 ] . In contrast, dimensional approaches quantify emotion along continuous axes, predominantly emotional arousal and valence [ 11 ] . Recent advances in large language models (LLMs) have expanded the scope of emotion recognition beyond the traditional paradigm of emotion label prediction. These models facilitate a more generative approach to emotion understanding , producing detailed, comprehensive descriptions of emotional states in natural language [ 27 ] . This shift has prompted the development of new datasets and metrics that accommodate rich natural language descriptions of emotions, allowing for greater nuance in emotion analysis. Many approaches to recognizing human emotions have used facial expressions [ 23 ] , speech or text sentiment. Increasingly, multimodal approaches that combine video, audio, text and image inputs are being explored as a way of improving model robustness in complex environments [ 26 ] . However, many existing multimodal models are still heavily reliant on [ 29 ] , or biased towards [ 47 ] , the language modality. To address this limitation, recent work has begun investigating novel sources of data that capture nonverbal social cues without relying on language, such as mime videos [ 25 ] . Sign language videos are also a rich source of expressive nonverbal data. They present unique challenges, but also opportunities, for multimodal emotion recognition. In hearing communities, many emotional expressions are universal in the sense that they can be reliably understood by people across many different cultures [ 10 ] . However, in sign languages, facial expressions and other non-manual components (such as mouth shapes and body language) often simultaneously serve grammatical and emotional functions. As such, recognizing the emotion of sign videos is often challenging for non-signers [ 30 ] . 2.2 Machine learning research on sign language In the context of machine learning research, work on sign language has focused on sign language translation and production. Sign language translation (SLT) methods typically use either raw image data or skeletal representation of the signer’s pose as input. In terms of model architectures, Transformer-based architectures have been used for word-level sign language recognition based

Read the complete text