MHB: Multimodal Handshape-aware Boundary Detection for Continuous Sign Language Recognition

Mingyu Zhao, Zhanfu Yang, Yang Zhou, Zhaoyang Xia, Can Jin, Xiaoxiao He, Carol Neidle, Dimitris N. Metaxas, [2pt] Rutgers University, Boston University · 2025

Read MHB: Multimodal Handshape-aware Boundary Detection for Continuous Sign Language Recognition by Mingyu Zhao, Zhanfu Yang, Yang Zhou, Zhaoyang Xia, Can Jin, Xiaoxiao He, Carol Neidle, Dimitris N. Metaxas, [2pt] Rutgers University, Boston University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19907.txt.

Introduction

Mingyu Zhao and six co-authors present MHB, a system for recognizing continuous American Sign Language. Their method first finds where each sign starts and ends in a sentence video, then recognizes the cut signs. It uses three-dimensional skeletal features, which capture how sign properties bunch near boundaries.

A handshape classifier, pretrained on eighty-seven linguistically defined handshape categories from a newly merged dataset, adds information about shapes expected at sign edges. A fusion module joins the segmentation and handshape models, and the found boundaries feed a recognition model trained on isolated and pre-cut signs. On the ASLLRP corpus, the method improves clearly over earlier work.

Catalog record

CREATORMingyu Zhao, Zhanfu Yang, Yang Zhou, Zhaoyang Xia, Can Jin, Xiaoxiao He, Carol Neidle, Dimitris N. Metaxas, [2pt] Rutgers University, Boston University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19907

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MHB: Multimodal Handshape-aware Boundary Detection for Continuous Sign Language Recognition Mingyu Zhao, Zhanfu Yang, Yang Zhou, Zhaoyang Xia, Can Jin, Xiaoxiao He, Carol Neidle, Dimitris N. Metaxas, [2pt] Rutgers University, Boston University Abstract This paper employs a multimodal approach for continuous sign recognition by first using ML for detecting the start and end frames of signs in videos of American Sign Language (ASL) sentences, and then by recognizing the segmented signs. For improved robustness we use 3D skeletal features extracted from sign language videos to take into account the convergence of sign properties and their dynamics that tend to cluster at sign boundaries. Another focus of this paper is the incorporation of information from 3D handshape for boundary detection. To detect handshapes normally expected at the beginning and end of signs, we pretrain a handshape classifier for detection of 87 linguistically defined canonical handshape categories using a dataset that we created by integrating and normalizing several existing datasets. A multimodal fusion module is then used to unify the pretrained sign video segmentation framework and handshape classification models. Finally, the estimated boundaries are used for sign recognition, where the recognition model is trained on a large database containing both citation-form isolated signs and signs pre-segmented (based on manual annotations) from continuous signing—as such signs often differ a bit in certain respects. We evaluate our method on the ASLLRP corpus and demonstrate significant improvements over previous work. 1 Introduction Signed languages are fully developed visual-gestural languages used by Deaf and hard-of-hearing communities worldwide. Unlike spoken languages, they convey meaning through hand shapes, orientations, movements, facial expressions, head gestures, and body posture [ 3 ] . American Sign Language (ASL) is the primary language used by more than 500,000 people in the United States [ 1 ] . It is also ranked as the third most studied non-native language in the country [ 18 ] . Despite advances in automatic speech and natural language processing, Sign Language Recognition (SLR) remains a complex problem for 2 main reasons: (1) the high-dimensional spatio-temporal dynamics of gestures, and (2) a general lack of large-scale annotated datasets [ 15 ] . Most research to date has focused on Isolated Sign Language Recognition (ISLR), where each input clip contains exactly one sign. However, in real-world communication, signed languages are produced as continuous streams of signing, not isolated signs. Compared with ISLR, Continuous Sign Language Recognition (CSLR) is more complex, in part because of significant differences in sign production in continuous signing as compared with citation forms [ 32 ] . Among many complicating factors in continuous signing are the potential independence of signing on the 2 hands, and the tendency for sign production to be influenced by the properties of adjacent segments. Note that in the current research, we are only attending to signs that include production by the dominant hand, i.e., 1-handed signs produced on the dominant hand and 2-handed signs. We are not considering 1-handed signs produced on the non-dominant hand. A major challenge in CSLR lies in the difficulty of locating precise temporal boundaries for each sign segment, i.e., identifying the start and end frames corresponding to each sign [ 32 ] . Another major difficulty arises from the fact that natural signing includes things that CANNOT just be looked up in a “dictionary” of isolated signs (even if they could be perfectly segmented)—including potentially: false starts, things like classifiers and gestures, which do not belong to any finite classes of elements, as well as fingerspelled signs, potentially corresponding to any word or proper name from the spoken language (often with letters missing or transposed). This is why any ultimate strategy for recognition of signs from continuous signing needs to be able to distinguish among types of signs, as different recognition strategies need to be applied to different types of signs [ 43 ] . Note, however, that the research reported here does not do this, so we have submitted our segmented signs for sign recognition only in cases where we have at least 6 examples of those signs in our training dataset. Thus, we are not currently carrying out end-to-end sign recognition, although that is the long-term goal. The sign recognition component of this research is intended to demonstrate the usefulness of segmentation for sign recognition. Some prior work relies on relatively small datasets for experimentation, such as [ 10 , 19 , 45 ] . Other work (e.g. [ 34 ] ) attempts to synthesize pseudo-continuous sign language datasets by concatenating isolated sign videos. However, none of these strategies has been applied to real large-scale continuous sign language datasets for boundary detection. Most approaches to sign boundary detection follow a similar 2-stage pipeline. They typically first extract visual features from raw videos using tools such as OpenPose or 3D convolutional neural networks, and then feed these features into deep learning models such as LSTMs or Transformers for temporal modeling and boundary prediction [ 34 , 36 , 19 ] . These methods have shown some success, but they generally neglect 2 important perspectives: (1) multimodal insights from patterns involving handshape, motion, velocity, acceleration, facial expressions, upper body position, and head gestures, known to correlate with sign transitions, which have the potential to provide strong signals for boundary detection; and (2) the use of more specialized skeleton-based approaches for pose modeling, as most methods rely on general-purpose deep models and overlook structured methods that are better suited to skeletal data. Building on these observations, we propose a multimodal boundary detection framework, incorporating a pretrained handshape classification model with a sign boundary detection model. We use the estimated boundaries for sign recognition in a well-trained sign recognition model. There is no standard written representation for signs. Researchers often use English-based ID glossing to identify signs, as we also do. But a major challenge is ensuring consistency in gloss labeling, i.e., a 1-1 correspondence between signs and glosses, and many datasets fail to meet this requirement (e.g., [ 28 ] ). One major advantage of the datasets we are using is that we ensure consistency in gloss representations across all the datasets we use. Our contributions are summarized as follows: • We design a segmentation module based on a spatiotemporal convolutional network, incorporating velocity and acceleration features. This outperforms state-of-the-art methods in CSL boundary detection. • We propose a novel method to enhance boundary detection by incorporating handshape information. A multimodal fusion module is introduced, which integrates features from a pretrained handshape classifier into the segmentation stream using a cross-attention mechanism with a gating mechanism. • We present a new CSL recognition pipeline combining a state-of-the-art segmentation model with a sign language recognition model. Experiments demonstrate the effectiveness of our segmentation framework. 2 Related Work 2.1 Isolated Sign Language Recognition Recent advances in Isolated Sign Language Recognition (ISLR) have leveraged both visual and skeletal modalities to recognize signs from trimmed video clips. Early methods mostly relied on handcrafted features and Hidden Markov Models (HMMs) [ 38 ] . With the rise of deep learning, convolutional neural networks (CNNs) [ 33 ] and hierarchical attention networks (HANs) [ 15 ] became widely used to model spatial and temporal dependencies in isolated signs. More recent approaches adopt BiLSTM

Read the complete text