Text-Driven Diffusion Model for Sign Language Production
Jiayi He Xu Wang Ruobei Zhang Shengeng Tang Yaxiong Wang Lechao Cheng · 2025
Read Text-Driven Diffusion Model for Sign Language Production by Jiayi He Xu Wang Ruobei Zhang Shengeng Tang Yaxiong Wang Lechao Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15914.txt.
Introduction
Jiayi He, Xu Wang, Ruobei Zhang, and a team of co-authors generate sign language poses from written text using a diffusion model. Translating text into signing is a first step toward tools that let deaf and hearing people communicate more easily. The work was the team's entry to a shared challenge on the task.
Their system encodes the text and feeds it as a condition into a diffusion process that builds the pose sequence joint by joint. Two penalty terms keep the output faithful: one matches joint positions with the reference signing, and the other matches the orientation of the bones. Starting from pure noise, the model refines its way to signing that fits the sentence. The entry placed second in the challenge.
Catalog record
| CREATOR | Jiayi He Xu Wang Ruobei Zhang Shengeng Tang Yaxiong Wang Lechao Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15914
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Text-Driven Diffusion Model for Sign Language Production Jiayi He Xu Wang Ruobei Zhang Shengeng Tang Yaxiong Wang Lechao Cheng Abstract We introduce the hfut-lmc team’s solution to the SLRTP Sign Production Challenge. The challenge aims to generate semantically aligned sign language pose sequences from text inputs. To this end, we propose a Text-driven Diffusion Model (TDM) framework. During the training phase, TDM utilizes an encoder to encode text sequences and incorporates them into the diffusion model as conditional input to generate sign pose sequences. To guarantee the high quality and accuracy of the generated pose sequences, we utilize two key loss functions. The joint loss function ℒ j o i n t \mathcal{L}_{joint} is used to precisely measure and minimize the differences between the joint positions of the generated pose sequences and those of the ground truth. Similarly, the bone orientation loss function ℒ b o n e \mathcal{L}_{bone} is instrumental in ensuring that the orientation of the bones in the generated poses aligns with the actual, correct orientations. In the inference stage, the TDM framework takes on a different yet equally important task. It starts with noisy sequences and, under the strict constraints of the text conditions, gradually refines and generates semantically consistent sign language pose sequences. Our carefully designed framework performs well on the sign language production task, and our solution achieves a BLEU-1 score of 20.17 20.17 , placing second in the challenge. 1 Introduction Figure 1 : The task of generating sign language pose sequences directly from text extremely daunting challenges. Classic pipelines for Sign Language Production: text-to-gloss, gloss-to-pose and pose-to-real video. In this work, we go beyond the commonly used pipeline and focus on the T2P task of generating sign pose sequences directly from text. Figure 2 : Overall framework of our method - TDM. It consists of a text encoder, a denoiser, and a MLP. The text encoder is primarily designed to capture the global semantics of the text sequence. This global semantics is integrated with the time step that has been processed by a MLP to form the relevant condition g . We add noise in t steps to the target pose sequence p_{0} to obtain p_{t} . Then, we send p_{t} along with the relevant condition g to the denoiser \mathcal{D} to recover the target pose sequence p_{0}^{\prime} that is free from noise contamination. For the optimization, we employed two main loss functions. The \mathcal{L}_{joint} loss function is primarily used to constrain the joint coordinates, while the \mathcal{L}_{bone} loss function is mainly applied to constrain the direction of bone movement. Sign language, an essential tool for seamless interaction between key research fields and the deaf community, is at the forefront of a technological revolution that significantly impacts the construction of information accessibility. In the digital age, with information flowing freely in most of society, ensuring the deaf can equally access this knowledge wealth is not only a matter of social justice but also a frontier in academic and technological exploration. Currently, academic research in sign language technology mainly focuses on three major interrelated directions: Sign Language Translation (SLT), Sign Language Recognition (SLR), and Sign Language Production (SLP). SLT aims to bridge the communication gap between sign language and spoken languages, facilitating effective cross-communication between the deaf and the hearing-impaired. SLR, in contrast, focuses on the automatic recognition of sign language gestures. Although significant progress has been made in SLT [ 30 , 29 , 4 , 22 , 7 ] and SLR [ 10 , 31 , 28 ] , SLP still poses a complex and urgent challenge. The complexity of SLP lies in the unique grammatical rules of sign language. Sign language is not merely a visual equivalent of spoken language; it is a separate language system with its own grammar, syntax, and semantics. Spoken language depends on auditory sounds and the temporal order of words, while sign language uses spatial positioning and gesture directionality to clearly show the relationships between sentence components. These spatial and gestural features are crucial to sign language grammar. For instance, the hand’s position in space can denote the subject, object, or location in a sign language sentence, and the gesture’s movement direction can express actions like moving towards or away from the signer. To address this challenge, existing methods generally adopt a two-step process. First, they convert text into gloss 1 1 1 Glosses differ from texts, with each gloss corresponding to a sign language. , a representation form that captures sign specific meaning. Then, sign language pose sequences are generated from this gloss. However, this approach has several shortcomings. The conversion from text to gloss and then to sign language poses adds multiple levels of complexity. It demands in-depth knowledge of both the source text language and sign language grammar, along with complex algorithms for the conversion. This complexity not only increases the computational load but also severely impacts the efficiency of SLP. In practical applications like real-time communication scenarios, this inefficiency can be a significant obstacle. Therefore, developing methods to generate sign language directly from text is not only highly urgent but also of crucial significance. These methods have the potential to revolutionize sign language production, making it more accessible, efficient, and user-friendly for the deaf community. To this end, we proposed a Text-Driven Diffusion Model (TDM) for sign language production. TDM employs a diffusion model architecture, encoding text sequences via an encoder and integrating them into the diffusion model as conditioning inputs for pose sequences generation. Meanwhile, we use bone orientation loss \mathcal{L}_{bone} to improve the quality of sign language pose sequences. Our main contributions are: • We introduce the diffusion model into the T2P task and use the skeleton constraint loss to generate results that are much more semantically rich than the baseline model. • In the SLRTP Sign Production Challenge, our solution achieved a BLEU-1 score of 22.17 and won the second place in the competition. The visualization results show that the results generated by our model are very close to the ground truth. 2 Related Work 2.1 Sign Language Production Early research in SLP mainly employed avatar-based methods [ 1 , 12 ] . However, due to the technological limitations that were prevalent at that time, the avatars showed rather jerky movements. For example, when an avatar was programmed to perform a simple walking motion, it would seem to lurch forward in an unnatural and disjointed way. Furthermore, their expressions were extremely rudimentary. In educational applications, an avatar designed to convey learning-related emotions might only be able to display a very general ”happy” or ”sad” expression, lacking the subtle nuances essential for effective communication. In the generation task, Variational Autoencoders (VAEs) [ 14 ] and Generative Adversarial Networks (GANs) [ 6 ] showed remarkable performance. Inspired by their performance, a series of methods based on Generative Adversarial Networks (GANs) [ 19 , 17 ] and Variational Autoencoders (VAEs) [ 11 , 26 ] have emerged. Although these methods try to control the mapping from text to visual representation by encoding text information, they often fail to handle complex semantics effectively. Take a sentence with multiple nested clauses and idiomatic expressions as an