Siformer: Feature-isolated Transformer for Efficient Skeleton-based Sign Language Recognition Conference: Proceedings of the 32nd ACM International Conference on Multimedia; October 28-November 1, 2024; Melbourne, VIC, Australia Proceedings of the 32nd ACM International Conference on Multimedia (MM

Muxin Pu, Mei Kuan Lim, Chun Yong Chong · 2025

Read Siformer: Feature-isolated Transformer for Efficient Skeleton-based Sign Language Recognition Conference: Proceedings of the 32nd ACM International Conference on Multimedia; October 28-November 1, 2024; Melbourne, VIC, Australia Proceedings of the 32nd ACM International Conference on Multimedia (MM by Muxin Pu, Mei Kuan Lim, Chun Yong Chong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20436.txt.

Introduction

Muxin Pu, Mei Kuan Lim, and Chun Yong Chong present a sign language reader in 2025 that works from body skeletons rather than raw video. Reading signs is hard because hands, posture, and face move fast and intricately. Skeleton input sidesteps background and appearance changes, but past methods used unrealistic hand poses and assumed complete data.

Their model first corrects hand skeletons with motion constraints, then reads each body feature stream on its own to stay robust when data go missing. An adaptive mode adjusts effort to the complexity of each sign. Tests show gains in accuracy and efficiency. The authors aim for readers that cope with real-world messiness.

Catalog record

CREATORMuxin Pu, Mei Kuan Lim, Chun Yong Chong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20436

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Siformer: Feature-isolated Transformer for Efficient Skeleton-based Sign Language Recognition Conference: Proceedings of the 32nd ACM International Conference on Multimedia; October 28-November 1, 2024; Melbourne, VIC, Australia Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24), October 28-November 1, 2024, Melbourne, VIC, Australia DOI: 10.1145/3664647.3681578 ISBN: 979-8-4007-0686-8/24/10 CCS: Computing methodologies Supervised learning by classification CCS: Computing methodologies Interest point and salient region detections CCS: Computing methodologies Feature selection CCS: Computing methodologies Neural networks CCS: Human-centered computing Human computer interaction (HCI) Muxin Pu, Mei Kuan Lim, Chun Yong Chong Abstract . Sign language recognition (SLR) refers to interpreting sign language glosses from given videos automatically. This research area presents a complex challenge in computer vision because of the rapid and intricate movements inherent in sign languages, which encompass hand gestures, body postures, and even facial expressions. Recently, skeleton-based action recognition has attracted increasing attention due to its ability to handle variations in subjects and backgrounds independently. However, current skeleton-based SLR methods exhibit three limitations: 1) they often neglect the importance of realistic hand poses, where most studies train SLR models on non-realistic skeletal representations; 2) they tend to assume complete data availability in both training or inference phases, and capture intricate relationships among different body parts collectively; 3) these methods treat all sign glosses uniformly, failing to account for differences in complexity levels regarding skeletal representations. To enhance the realism of hand skeletal representations, we present a kinematic hand pose rectification method for enforcing constraints. Mitigating the impact of missing data, we propose a feature-isolated mechanism to focus on capturing local spatial-temporal context. This method captures the context concurrently and independently from individual features, thus enhancing the robustness of the SLR model. Additionally, to adapt to varying complexity levels of sign glosses, we develop an input-adaptive inference approach to optimise computational efficiency and accuracy. Experimental results demonstrate the effectiveness of our approach, as evidenced by achieving a new state-of-the-art (SOTA) performance on WLASL100 and LSA64. For WLASL100, we achieve a top-1 accuracy of 86.50%, marking a relative improvement of 2.39% over the previous SOTA. For LSA64, we achieve a top-1 accuracy of 99.84%. The artefacts and code related to this study are made publicly online. 1 1 1 https://github.com/mpuu00001/Siformer.git 1. Introduction Sign languages (SL) ( Emmorey 2001 ) were invented to facilitate natural and intuitive communication, aiming to assist people with hearing or speech impairments for a better quality of life. Globally, there are around 466 million people with these disabilities and more than 300 SLs ( WHO 2024 ) . Mastering SL proficiency remains a challenge for the general public since it is deeply influenced by the corresponding spoken language ( Valli and Lucas 2000 ; Johnston and Schembri 2007 ; Yang 2010 ) and the associated culture ( Mindess 2014 ) . In light of significant advancements in machine learning, it is crucial to delve into SLR to facilitate smooth communication for those with hearing or speech impairments. In this work, we focus on isolated SLR, a key task in visual SL research, which aims to recognise SL at the word level and is a fine-grained classification problem. Recently, skeleton-based methods have gained popularity in the SLR task ( Boháček and Hrúz 2022 ; Tunga et al. 2021 ; Li et al. 2020 ) due to their computational efficiency and portability. These methods analyse dynamic patterns from sequences of body poses, represented by skeletal joints and facial landmarks extracted from pose estimators, to effectively recognise sign glosses, which are words associated with the signs. Pose estimation becomes crucial in this context to measure the position and movement of individual fingers, the hand, and other related features. However, these methods still encounter three major limitations in the realm of skeleton-based SLR: Firstly, non-realistic skeletal representation is a key problem that is often overlooked in SLR research. The more realistic the features and patterns that the chosen pose estimator can extract, the deeper the understanding can be gained in the context of SLR. SOTA pose estimators suffer from dynamic circumstances or complicated backgrounds, where issues such as occlusion, illumination variations, and motion blur come into play from time to time ( Pu et al. 2023 ) , leading to a further decrease in the overall anatomical correctness, the precision and realism of the output skeletal representation. This problem serves as the root cause of the unstable performance of skeleton-based SLR models, which are trained on misleading information. Secondly, missing data is another common problem in practice that has not been well explored in SLR research. Typical machine learning models assume complete data availability in both the training and inference phases. However, in reality, there is a high risk that certain data may be absent at either phase due to malfunctions during data collection, high data acquisition costs, human errors in data preprocessing, etc ( Woo et al. 2023 ) . For instance, the joints of human hands are prone to occlusion or self-occlusion due to articulation or viewpoints. The hands, body parts, and faces may fail to be detected under the influence of motion blur and illumination variations ( Pu et al. 2023 ) . As such, a mechanism that enhances SLR model robustness to missing data is essential. Thirdly, inefficient inference is a problem generally caused by overthinking during the decision-making process. For many input samples, the shallow representations at an earlier layer are sufficient to make a correct classification, whereas the representations at the final layer may become distracted by over-complicated or irrelevant features that do not generalise well ( Zhou et al. 2020 ) . The complexity level of the sign glosses varies in SL. Some sign glosses need a mix of global body movements, delicate gestures performed by the highly articulated hand, and (micro-)facial expressions. In contrast, some sign glosses can be easily performed by a single hand. Existing SLR models typically handle all sign glosses similarly, regardless of their complexity level in terms of skeletal representations. This inefficiency may limit the deployment of SLR models in contexts where inference speed and computational costs are critical, as well as making them vulnerable to adversarial attacks. To address the aforementioned limitation, this paper proposes the following solutions: (1) We present a kinematic hand pose rectification approach that implemented constraints on the hand joints based on their kinematic constraints, since hand gestures play a dominant role in performing SL. These constraints derived from ( Hochschild 2016 ; Ross et al. 2006 ; Isaac et al. 2022 ) define the allowable range of motion for each finger’s flexion-extension and abduction-adduction movements. This approach enhances the realism of hand skeletal representations, resulting in more reasonable and lifelike representations of SL glosses by skeletal data. (2) We propose a feature-isolated mechanism to handle hand gestures and body gestures separately, improving efficiency in attention calculation and easing the impact of detection failures of hands and other body parts. Based on our observations, detection failures of hands and other body parts are independent. They are usually estimated by different off-the-shelf estimators. The hand occupies a relatively smaller area, making the detection failure rates higher than other body parts. Commonly, Transformers rely heavily on the attention mechanism ( Vaswani et al. 2017 ) to

Read the complete text