AutoSign: Direct Pose-to-Text Translation for Continuous Sign Language Recognition
Samuel Ebimobowei Johnny, Blessed Guda, Andrew Blayama Stephen, Assane Gueye · 2025
Read AutoSign: Direct Pose-to-Text Translation for Continuous Sign Language Recognition by Samuel Ebimobowei Johnny, Blessed Guda, Andrew Blayama Stephen, Assane Gueye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19840.txt.
Introduction
Samuel Ebimobowei Johnny and co-authors translate sign language straight from body pose to text, in work from 2025. Continuous signing blends hand, face, and body motion into one stream. Standard pipelines split the job into stages, and errors then multiply across stages.
Their system, AutoSign, uses a decoder-only transformer fed with pose data. It writes plain language text in one pass, skipping the gloss step entirely. Dropping the alignment stage also removes a known scaling limit on vocabulary. Training treats signing as direct translation between two forms. The design aims at smoother, less brittle reading of real signing. Tests target continuous sequences rather than single signs.
Catalog record
| CREATOR | Samuel Ebimobowei Johnny, Blessed Guda, Andrew Blayama Stephen, Assane Gueye |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19840
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AutoSign: Direct Pose-to-Text Translation for Continuous Sign Language Recognition Samuel Ebimobowei Johnny, Blessed Guda, Andrew Blayama Stephen, Assane Gueye Abstract Continuously recognizing sign gestures and converting them to glosses plays a key role in bridging the gap between the hearing and hearing-impaired communities. This involves recognizing and interpreting the hands, face, and body gestures of the signer, which pose a challenge as it involves a combination of all these features. Continuous Sign Language Recognition (CSLR) methods rely on multi-stage pipelines that first extract visual features, then align variable-length sequences with target glosses using CTC or HMM-based approaches. However, these alignment-based methods suffer from error propagation across stages, overfitting, and struggle with vocabulary scalability due to the intermediate gloss representation bottleneck. To address these limitations, we propose AutoSign, an autoregressive decoder-only transformer that directly translates pose sequences to natural language text, bypassing traditional alignment mechanisms entirely. The use of this decoder-only approach allows the model to directly map between the features and the glosses without the need for CTC loss while also directly learning the textual dependencies in the glosses. Our approach incorporates a temporal compression module using 1D CNNs to efficiently process pose sequences, followed by AraGPT2, a pre-trained Arabic decoder, to generate text (glosses). Through comprehensive ablation studies, we demonstrate that hand and body gestures provide the most discriminative features for signer-independent CSLR. By eliminating the multi-stage pipeline, AutoSign achieves substantial improvements on the Isharah-1000 dataset, achieving an improvement of up to 6.1% in WER score compared to the best existing method. \setcode utf8 1 Introduction Figure 1 : Overview of the proposed DTrOCR method. With the help of the transformer decoder, temporal pose sequences are directly translated to Arabic text without intermediate gloss supervision. This makes the model focus more on end-to-end optimization and cross-modal alignment. To explore the temporal dynamics effectively, a configurable 1D CNN compression is proposed in the pose encoding module. The deaf and hard-of-hearing community, comprising about 5.5% of the global population [ 42 ] , relies on Sign Language (SL) as their primary mode of communication. Unlike spoken languages, sign languages convey information through a rich combination of hand gestures, facial expressions, and body movements, creating complex visual-spatial languages with separate grammar and vocabulary structures. This intrinsic complexity creates significant challenges for people who are unfamiliar with sign language, limiting effective communication and social inclusion. While glosses remain the fundamental unit of sign language representation, the difference in sign languages across regions and groups presents a significant challenge for widespread adoption of communication technologies. With over 200 distinct sign languages across the world, and each possessing unique gesture signs, it becomes practically impossible for individuals to master multiple SLs. Furthermore, it is unlikely that people without such impairments will learn an additional language, which is not seen as a necessity for them [ 41 ] , further emphasizing the critical need for automated translation systems. Continuous Sign Language Recognition systems have been primarily developed for popularly used sign languages such as American Sign Language (ASL), Chinese Sign Language (CSL), and German Sign Language (GSL) [ 31 ] .In this work, we focus on developing a robust CSLR system for Saudi Sign Language (SSL) and assessing its performance on unseen signers not included during training. Standard CLSR methods use a multi-stage three-module approach: spatial feature extraction, temporal modeling, and sequence alignment components. These methods typically use Connectionist Temporal Classification (CTC) loss or Hidden Markov Models (HMMs) to map different sequence lengths to the corresponding gloss length, which increases complexity, introduces overfitting, and usually works well for smaller vocabularies [ 6 ] . To address these limitations, we present a novel method that directly translates pose sequences to natural language text without the need for intermediate gloss supervision. In contrast to traditional multi-stage pipelines, our method employs a decoder-only transformer architecture that performs end-to-end autoregressive generation from pose inputs to Arabic text outputs, as illustrated in Figure 1 . This approach eliminates the need for complex alignment mechanisms and gloss annotations while leveraging the robust sequential modeling capabilities of transformer architectures. Inspired by the success of decoder-only models in language modeling [ 36 , 38 ] , we demonstrate that autoregressive generation naturally extends to the cross-modal task of pose-to-text translation. Our approach incorporates a configurable temporal compression module using 1D CNNs to efficiently process long pose sequences while preserving important temporal dynamics. Through comprehensive experiments on the Isharah Arabic Sign Language dataset, we demonstrate that direct pose-to-text learning can achieve competitive performance while simplifying the typical CSLR pipeline. Furthermore, we conduct detailed ablation studies analyzing the contribution of different body parts (hands, face, body) to recognition performance, providing insights into optimal pose representations for sign language understanding. 2 Related Work 2.1 Continuous Sign Language Recognition Sign language recognition can be categorized into two group: isolated sign language recognition (ISLR) [ 18 , 17 , 37 , 34 , 24 ] and continuous sign language recognition (CSLR) [ 33 , 15 , 19 , 22 , 21 , 5 , 1 ] . Unlike ISLR, which focuses on recognizing individual signs with clear temporal boundaries, CSLR addresses the more challenging task of recognizing continuous sign sequences without prior segmentation, making it essential for real-world applications. Traditional CSLR architectures usually have three main parts: spatial feature extraction, temporal modeling, and sequence alignment [ 6 ] . The spatial module extracts frame-level features using 2D-CNNs [ 10 , 2 , 45 ] , 3D-CNNs [ 1 , 14 ] , graph convolutional networks (GCNs) [ 45 , 40 ] , or transformers [ 3 , 44 ] . The temporal module captures sequential dependencies through 1D-CNNs [ 14 , 21 , 22 , 25 ] , spatio-temporal GCNs [ 35 , 16 ] , RNNs [ 14 , 25 , 23 ] , or transformers [ 48 , 5 ] . Finally, the alignment module maps variable-length video sequences to gloss sequences using connectionist temporal classification (CTC) [ 7 , 23 , 11 , 1 , 45 ] , correlation maps [ 20 , 6 ] , reinforcement learning [ 4 ] , or hidden Markov models (HMMs) [ 28 , 27 ] . Recent studies have found major problems with these alignment methods: CTC-based approaches often overfit the data and suffer from error propagation across stages, while HMM-based methods typically need to be trained on small vocabularies, which makes them less flexible and harder to apply broadly [ 6 ] . This three-stage module has become the dominant paradigm in CSLR research, with the most recent focus on improving the alignment components. 2.2 Pose-based Sign Language Recognition Most CSLR methods [ 33 , 15 , 19 , 22 , 5 , 21 , 1 ] primarily used RGB video data, but recent approaches have transitioned to pose-based representations [ 39 , 12 , 25 , 47 , 6 ] derived from frameworks like MediaPipe [ 30 ] and OpenPose [ 9 ] . The change is driven by several advantages: pose data offers robustness against background variations, lighting circumstances, and clothing discrepancies while preserving critical spatial-temporal information regarding sign articulation. Additionally, posture representations substantially reduce computational overhead compared to raw RGB processing. Pose-based models typically extract