Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production
Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, Richang Hong* · 2024
Read Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production by Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, Richang Hong* (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13609.txt.
Introduction
Shengeng Tang and five co-authors presented Sign-IDD in two thousand twenty-four, a method to turn text into sign language poses. Making sign videos from text hinges on a step that turns gloss words into body poses. Past methods fit joint points directly and miss how joints relate to each other, which can make poses look stiff or wrong.
Their method splits each pose into bone data: the direction and length between neighbouring joints. A diffusion model then builds poses under the guidance of those bone traits and the gloss meaning. Tests on two sign data sets, German and Chinese, show the poses are more exact and natural than past methods.
Catalog record
| CREATOR | Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, Richang Hong* |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13609
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, Richang Hong* Abstract Sign Language Production (SLP) aims to generate semantically consistent sign videos from textual statements, where the conversion from textual glosses to sign poses (G2P) is a crucial step. Existing G2P methods typically treat sign poses as discrete three-dimensional coordinates and directly fit them, which overlooks the relative positional relationships among joints. To this end, we provide a new perspective, constraining joint associations and gesture details by modeling the limb bones to improve the accuracy and naturalness of the generated poses. In this work, we propose a pioneering iconicity disentangled diffusion framework, termed Sign-IDD, specifically designed for SLP. Sign-IDD incorporates a novel Iconicity Disentanglement (ID) module to bridge the gap between relative positions among joints. The ID module disentangles the conventional 3D joint representation into a 4D bone representation, comprising the 3D spatial direction vector and 1D spatial distance vector between adjacent joints. Additionally, an Attribute Controllable Diffusion (ACD) module is introduced to further constrain joint associations, in which the attribute separation layer aims to separate the bone direction and length attributes, and the attribute control layer is designed to guide the pose generation by leveraging the above attributes. The ACD module utilizes the gloss embeddings as semantic conditions and finally generates sign poses from noise embeddings. Extensive experiments on PHOENIX14T and USTC-CSL datasets validate the effectiveness of our method. The code is available at: https://github.com/NaVi-start/Sign-IDD. School of Computer Science and Information Engineering, Hefei University of Technology {tangsg, guodan, weiyy, fengli, hongrc}@hfut.edu.cn, [email protected] Figure 1: (a) Problems caused by using only traditional 3D representations for SLP. (b) Traditional 3D joint coordinate representation ( Saunders, Camgoz, and Bowden 2020b ; Tang et al. 2022b ) . (b) Proposed 4D bone representation. We take the neck joint as the root joint and define the parent-child joints for each bone along the skeletons. The Euclidean distance and direction vectors between parent-child joints are used to determine the length and orientation of the bones. Introduction Sign Language Production (SLP) plays a crucial role in bridging the communication gap between the deaf and the general population, promoting inclusion and accessibility. This task is technically closely related to areas such as visual understanding ( Wei et al. 2024 ; Li et al. 2024b ; Guo et al. 2024 ; Li et al. 2024a ) and cross-media reasoning ( Song et al. 2024 ; Wu, Hong, and Tang 2024 ; Song et al. 2023 ) . Given a textual description, SLP aims to transform it into the corresponding sequence of continuous signs automatically. These sequences can manifest as sign language poses ( Saunders, Camgoz, and Bowden 2020a ; Saunders, Camgoz, and Bowden 2020b ) or sign language videos ( Saunders, Camgoz, and Bowden 2022a ) . Currently, direct sign language video generation from spoken sentences remains a challenge due to the huge gap between sign vision and linguistics. Previous works usually translate spoken language into gloss 1 1 1 Glosses refer to minimal lexical items that match the meaning of signs in linguistics. sequence (T2G) first and then generate sign pose video (G2P) based on gloss sequence ( Saunders, Camgoz, and Bowden 2020a ; Saunders, Camgoz, and Bowden 2020b ) . Finally, the produced gesture poses are selectively used to generate realistic gesture videos ( Saunders, Camgoz, and Bowden 2022a ) . Since T2G can be well addressed by Neural Machine Translation (NMT, language-to-language) based ( Othman and Jemni 2011 ) and rule-based approaches ( Moryossef et al. 2021 ) , G2P remains the key procedure for this task at this stage and is the focus of this work. Depending on the decoding strategy, current G2P methods are typically classified as either autoregressive ( Saunders, Camgoz, and Bowden 2020a ; Saunders, Camgoz, and Bowden 2020b ; Tang et al. 2022b ) or non-autoregressive ( Huang et al. 2021 ; Xie et al. 2024 ; Tang et al. 2024 ) . These efforts have promoted the development of SLP tasks, especially G2P-DDM ( Xie et al. 2024 ) and GCDM ( Tang et al. 2024 ) as representative diffusion-based solutions, which improve the accuracy of generating sign poses. However, existing methods typically treat sign poses as discrete three-dimensional coordinates and only focus on the regression prediction of joint coordinates. These solutions overlook the modeling of relative positional relationships among joints, which hinders the effective generation of sign poses. Our goal is to generate clear and accurate gestures, especially in terms of poses that affect the semantic expression of sign language. In the example of Figure 1 (a), most of the joints in the output are already close to the target distribution in spatial position. However, their relative positions are quite different from the facts, especially in the finger details that are susceptible to deviation (yellow box). In addition, the orientation of the generated limbs also shows unexpected deviations (green box). A better constraint of bone details and limb orientation should be established in the generated pose using relative positions among the joints. Although some methods can deal with skeletal constraints by employing graph models ( Saunders, Camgoz, and Bowden 2022b ) , they do not consider the inherent skeletal supervision and lead to higher computational costs. Fortunately, in the field of pose estimation, some emerging works have considered and verified the positive impact of constraining human bone length and orientation directly on prediction performance ( Cai et al. 2024 ) . Inspired by these works, we introduce an iconicity disentanglement strategy to enhance the relative position association (including orientations and distances) among joints. To this end, we propose a novel iconicity disentanglement diffusion framework, Sign-IDD, which aims to improve the expressiveness and accuracy of sign language gestures by enhancing the spatial association among joints. Sign-IDD incorporates a novel Iconicity Disentanglement (ID) module, to improve the perception of relative positions among joints. As shown in Figure 1 (b) and (c), two adjacent joints along each bone are regarded as parent-child joints. The ID module is designed to disentangle the conventional 3D joint representation into a 4D bone representation, comprising the 3D spatial direction vector and 1D spatial distance vector from parent to child joint. As shown in Figure 2 , we further construct a diffusion-based SLP framework, where gloss embeddings are used as semantic conditions to guide sign language gesture generation. In this framework, the Attribute Controllable Diffusion (ACD) module is another core component, which further strengthens the association learning of joints in the generated poses. The ACD module first incorporates gloss conditions into the pose embeddings to achieve semantic guidance. The attribute separation layer aims to separate the bone direction and length attributes, and the attribute control layer is designed to optimize the sign generation under the guidance of the above attributes. Our main contributions are summarized as follows: • We innovatively introduce the concept of iconicity disentanglement, a novel strategy that transcends traditional joint coordinate regression fitting. Unlike most previous works that only adopt 3D joint coordinate representation, we utilize the disentangled 4D bone representation to further constrain the relative positions of joints, thereby ensuring the accuracy of sign pose details. • We further propose a novel