SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition

Kunyuan Xie, Zhixi Cai, Kalin Stefanov · 2026

Read SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition by Kunyuan Xie, Zhixi Cai, Kalin Stefanov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02094.txt.

Introduction

Kunyuan Xie, Zhixi Cai, and Kalin Stefanov improve sign language recognition by pretraining an encoder to notice hands and body parts in motion. Small hand differences change a sign's meaning, yet many encoders start from generic action data that miss those fine cues. Their method, SignMAE, masks parts of the input based on segmentation of key body regions.

The model then learns by rebuilding the masked parts, which sharpens its sense of fine sign features. On the WLASL, NMFs-CSL, and Slovo data sets, the encoder reached the best reported results for both per-instance and per-class top-one accuracy. It did so while using fewer input frames and fewer input types than comparable encoders.

Catalog record

CREATORKunyuan Xie, Zhixi Cai, Kalin Stefanov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02094

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition Kunyuan Xie, Zhixi Cai, Kalin Stefanov Abstract Subtle hand differences make sign language recognition challenging, yet many existing methods rely on encoders pretrained on generic action datasets that poorly capture such fine-grained cues. We propose a self-supervised pretraining method for sign language recognition that uses segmentation-based masking to adapt to the presence and motion of key body parts, rather than treating hand poses as static visual tokens. The resulting mask-and-reconstruct objective improves fine-grained sign representation learning. On WLASL, NMFs-CSL, and Slovo, our encoder achieves state-of-the-art performance, improving per-instance and per-class Top-1 accuracy while using fewer input frames and modalities than comparable encoders. 1 Introduction Sign languages are visual-gestural languages used by Deaf communities worldwide. Unlike spoken languages, they convey meaning through coordinated movements of the hands, body, and face. Isolated sign language recognition (ISLR) classifies short video clips into single-word glosses and underpins downstream tasks such as sign spotting, retrieval, translation, and continuous sign language recognition. In ISLR, the hands carry most of the linguistic signal, yet they occupy only a small portion of each frame and appear under varying backgrounds and clothing. Existing methods mainly use raw video or keypoints extracted from video. Skeleton-based approaches model hand keypoints alone or together with body keypoints, but standard keypoint extractors often fail, and keypoint representations omit fine-grained visual cues. Although self-supervised mask-and-reconstruct methods have been applied to sign keypoints [ 28 , 7 ] , video-based models still outperform them. Most video-based ISLR methods [ 29 , 16 , 6 ] use backbones pretrained for generic action recognition, which process the entire frame and may under-emphasize the hands. VideoMAEv2 [ 25 ] demonstrates the effectiveness of self-supervised video pretraining by masking spatio-temporal tubes and reconstructing missing content. However, its random tube masking is not well suited to sign language, since many sampled tubes contain little useful hand information. Despite recent progress, ISLR remains challenging. On WLASL [ 16 ] , which contains 2,000 glosses, top-1 accuracy is still only around 60%. This gap motivates more effective hand-centric representation learning. To this end, we propose SignMAE, a segmentation-guided self-supervised framework that learns complementary representations from video and keypoints. During pretraining, modality-specific encoder-decoder backbones reconstruct only tokens from hand regions, encouraging attention to linguistically salient cues. We further design masking strategies tailored to the hand regions. For downstream ISLR, the video and keypoint encoders are finetuned separately and fused through cross-attention. Our contributions are as follows: • We propose SignMAE, a self-supervised framework that learns complementary hand-centric representations from video and keypoints for ISLR. • We introduce segmentation-guided masking strategies that reconstruct only hand-region tokens during pretraining, encouraging both encoders to focus on linguistically salient information. • We validate SignMAE on NMFs-CSL [ 9 ] , Slovo [ 12 ] , and WLASL [ 16 ] , achieving state-of-the-art performance with fewer input frames and modalities than prior methods. 2 Related Work Early isolated sign language recognition methods typically use either keypoints or video alone. Keypoint-based approaches, including ST-GCN variants and more recent self-supervised pose models, capture spatio-temporal structure efficiently but are limited by keypoint detection errors and the loss of fine-grained visual detail [ 7 , 28 , 27 , 15 , 2 ] . Video-based methods learn directly from RGB frames using 2D CNN-RNN pipelines, 3D CNNs, or transformer-style backbones, and generally achieve stronger performance, though most rely on architectures developed for generic action recognition rather than sign-specific representation learning [ 16 , 3 , 6 , 9 ] . Multi-modal methods combine video and keypoints, or additional modalities such as optical flow and depth, to improve robustness, but often require longer input clips, heavier architectures, or more modalities during inference [ 24 , 7 , 11 , 29 ] . More broadly, masked autoencoding has become an effective self-supervised paradigm for visual representation learning: VideoMAE [ 23 ] extends this idea to videos with random tube masking, and VideoMAEv2 [ 25 ] improves efficiency with a stronger training recipe and partial masked-token decoding. However, such generic masking strategies are not optimized for sign language, where hands occupy only a small portion of the frame. This motivates our segmentation-guided, hand-centric pretraining method, which focuses reconstruction on linguistically informative hand and arm regions while learning complementary video and keypoint representations for ISLR. 3 Methodology Our pipeline proceeds through four sequential stages: 1) Data preprocessing extracts important patches that contain the hand regions for downstream learning; 2) Uni-modal self - supervised pretraining is performed independently on each modality: two video streams and one keypoint stream; 3) Uni - modal finetuning likewise treats the two modalities separately: the reconstruction decoder is replaced by a classification head, and each encoder is supervised to predict the gloss label of its input video; and 4) Multi - modal fusion uses the fine-tuned frozen encoders and integrates their representations with a cross - attention module, allowing joint perception over appearance and motion. The effectiveness of SignMAE, like other ISLR approaches, depends on the quality of intermediate outputs produced by off-the-shelf detectors for bounding boxes, segmentation, and keypoints. Challenging conditions, such as self-occlusion, low lighting, or motion blur, can degrade these signals and consequently affect downstream performance. This limitation is not unique to our framework: all prior ISLR methods that incorporate keypoint extraction similarly rely on pretrained models for these steps. Importantly, our pipeline is designed to maximise the signal-to-noise ratio for pretraining and finetuning, not to eliminate every possible imperfection. Small, isolated inaccuracies from the detectors typically introduce negligible noise relative to the full sequence, and the global motion pattern of a sign remains intact. As a result, minor upstream errors rarely compromise the quality of the learned representations, while broadly reliable preprocessing remains sufficient for SignMAE to achieve strong performance. More details on the data preprocessing pipeline are provided in the Supplementary Material. Figure 1: Illustration of the data preprocessing pipeline for our SignMAE framework. The data preprocessing pipeline extracts important patches and checks hand movement based on body segments and keypoints extracted from pretrained models. 3.1 Data Preprocessing To maximise the signal-to-noise ratio for later stages, we first standardize raw videos and crop patches that contain the hand regions. As shown in Fig. 1 , the input to the pipeline is a sign video with any dimensions. First, a detection model [ 22 ] predicts the signer’s bounding box in all frames, and then the frames are cropped. The output of this step is a cropped video with the same height and width. Next, we leverage a pretrained body segmentation model [ 19 ] and a keypoint detection model [ 14 ] to extract body segments and keypoints separately. We noticed that some frames at the beginning and end of most videos do not contain useful information because both hands are hanging naturally or do not exist. Therefore, we remove those frames to improve the efficiency of the mask-and-reconstruct pretraining. Specifically, we check the following parameters to determine whether an arm hangs naturally in a given frame: 1) Angle a_{1} formed by the left shoulder, right shoulder, and elbow; 2) Angle a_{2} formed by the shoulder, elbow, and wrist on the same side of the body; 3) Distance

Read the complete text