Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysis

Rui Hong, Jana Košecká · 2026

Read Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysis by Rui Hong, Jana Košecká (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17388.txt.

Introduction

Hong and Kosecka train a diffusion model that turns text into 3D sign language motion. They test whether phonological cues from ASL-LEX 2.0, like hand shape and location, help. Their baseline beat a prior CVAE method on gloss clarity scores. Turning symbolic attribute notes into plain words proved key for CLIP-based conditioning. The best variant beat the prior method on all metrics, they report.

Catalog record

CREATORRui Hong, Jana Košecká
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28815
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.17388

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysis Rui Hong, Jana Košecká Abstract Generating natural, correct, and visually smooth 3D avatar sign language motion conditioned on the text inputs continues to be very challenging. In this work, we train a generative model of 3D body motion and explore the role of phonological attribute conditioning for sign language motion generation, using ASL-LEX 2.0 annotations such as hand shape, hand location and movement. We first establish a strong diffusion baseline using an Human Motion MDM-style 23 diffusion model with SMPL-X representation, which outperforms SignAvatar 7 , a state-of-the-art CVAE method, on gloss discriminability metrics. We then systematically study the role of text conditioning using different text encoders (CLIP vs. T5), conditioning modes (gloss-only vs. gloss+phonological attributes), and attribute notation format (symbolic vs. natural language). Our analysis reveals that translating symbolic ASL-LEX notations to natural language is a necessary condition for effective CLIP-based attribute conditioning, while T5 is largely unaffected by this translation. Furthermore, our best-performing variant (CLIP with mapped attributes) outperforms SignAvatar across all metrics. These findings highlight input representation as a critical factor for text-encoder-based attribute conditioning, and motivate structured conditioning approaches where gloss and phonological attributes are encoded through independent pathways. Code is available at https://github.com/hongrui16/phonology-guided-sign-gen . 1 Introduction Sign language is the primary mode of communication for millions of deaf or hard-of-hearing individuals worldwide. Automatic generation of natural-looking and correct sign language motion has broad applications in accessible communication, virtual signing avatars, and sign language education. This, however, remains a challenging problem due to the complex movement of hand shapes at different hand locations, palm orientation, as well as co-articulated face expressions. Recent work on sign language generation has made progress using generative models conditioned on gloss labels 7 . However, these approaches typically ignore the internal phonological structure that linguists have long recognized as fundamental to sign language organization 22 ; 4 . Just as spoken language phonology decomposes words into phonemes, sign language phonology decomposes signs into a small set of articulatory parameters: hand shape, location, movement, palm orientation, and non-manual signals. The ASL-LEX 2.0 database 20 provides these structured annotations for over 2,700 ASL signs across 22 phonological dimensions. Incorporating phonological structure into sign generation offers compelling advantages. First, it enables compositional generalization : a model that understands individual phonological dimensions can potentially generate novel sign-like motions by recombining known attributes. Second, it provides interpretable control : users or downstream systems could independently adjust handshape, location, or movement parameters. Third, it introduces a linguistically grounded inductive bias that could improve data efficiency by sharing information across signs with similar phonological profiles. In this paper, we investigate phonological attribute conditioning for 3D sign language motion generation within a diffusion-based framework. Our contributions are: 1. We establish a strong diffusion baseline for gloss-conditioned sign motion generation using an MDM-style 23 Transformer Encoder denoiser operating on SMPL-X 14 upper-body motion in 6D rotation representation 25 . This baseline outperforms SignAvatar 7 , a state-of-the-art CVAE method, on gloss discriminability while achieving comparable distributional quality; with mapped phonological attribute conditioning, our model further surpasses SignAvatar across all reported metrics. 2. We conduct a systematic study of phonological attribute conditioning for sign motion generation via an ablation over text encoders (CLIP 16 vs. T5 17 ), conditioning modes (gloss-only vs. gloss+phonological attributes from ASL-LEX 2.0), and attribute notation format (symbolic vs. natural language). We demonstrate that translating symbolic ASL-LEX notations to natural language is a necessary condition for effective CLIP-based attribute conditioning, and that the two encoders respond asymmetrically to this translation. The ablation of the conditioning e The findings presented here motivate the use of phonological attributes as conditioning signals, suggesting a principled direction for phonology-guided generation. 2 Related Work Sign Language Generation. Sign language generation aims to produce natural signing motion from linguistic input. Early approaches used rule-based systems 10 that synthesized motion using concatenation of primitives. More recent data-driven approaches use deep learning methods to train generative models of realistic avatar motion conditioned on text or gloss with increasing naturalness. SignAvatar 7 uses a conditional variational autoencoder (CVAE) conditioned on CLIP embeddings of gloss labels to generate SMPL-X motion parameters. 2 train a diffusion model with an anatomically-informed GNN encoder and autoregressive LSTM decoder, conditioned on frozen CLIP sentence embeddings, on a large-scale 3D-annotated version of How2Sign to generate continuous sentence-level ASL motion from free-form text. Ham2Pose 1 generates pose sequences from HamNoSys notation. Progressive Transformer 18 and its extensions 19 address sentence-level sign language production from text. However, these methods do not exploit internal structure, or instead rely on transcription systems (HamNoSys) that require expert annotation. Our work bridges this gap by conditioning generation on phonological attributes from ASL-LEX 2.0, providing a structured linguistic description for a large gloss vocabulary. Human Motion Diffusion Models. Diffusion models have achieved strong results in human motion generation. Human Motion Diffusion Model MDM 23 uses a Transformer Encoder denoiser with classifier-free guidance for text-conditioned motion generation. MotionDiffuse 24 applies diffusion to body-part-level motion control. Motion Latent-based Diffusion Model MLD 6 introduces a latent diffusion approach for efficient motion generation. For hand and sign motion specifically, 3 train a text-conditioned hand motion diffusion model (HandMDM) on large-scale sign language data, where sign categories are automatically translated into natural language motion descriptions via an LLM, demonstrating cross-domain generalization to unseen sign categories and cross-lingual signs. Our work builds on the MDM architecture but focuses specifically on sign language generation with structured phonological conditioning, and provides a systematic analysis of conditioning mechanisms. Sign Language Phonology and ASL-LEX. Sign language phonology, pioneered by Stokoe 22 and refined by Brentari 4 , decomposes signs into simultaneous articulatory parameters. ASL SignBank 9 is a widely used community resource that documents a large vocabulary of ASL signs with video examples and categorical linguistic annotations. The original ASL-LEX database 5 introduced a quantitative lexical resource for ASL combining behavioral frequency measures with phonological coding. ASL-LEX 2.0 20 substantially expanded this to 2,723 ASL signs annotated across 22 phonological dimensions including handshape, selected fingers, flexion, thumb position, sign type, major location, minor location, path movement, and others. While ASL SignBank covers a larger vocabulary, ASL-LEX 2.0 provides richer phonological attributes per sign, making it more suitable as a conditioning signal for generative models. These annotations have been widely used in psycholinguistic research but remain underexplored in sign language generation. This work represents, to the best of our knowledge, the first systematic study of ASL-LEX phonological attributes as conditioning inputs for 3D motion generation. 3 Method We present a diffusion-based framework for sign language motion generation conditioned on gloss labels and, optionally, phonological attributes. We train a transformer-based diffusion model on a subset of the WLASL dataset with gloss annotations and explore different strategies for text conditioning. Figure 1 provides an overview of the proposed model. 3.1 Motion Representation Similarly to previous works, we represent the articulated human body using SMPL-X 14 body model parameters. Since sign language primarily involves the upper body, we operate on a subset of 43 upper-body joints (excluding the root, lower body, and jaw), comprising the spine, torso, arms, and both hands. Each joint rotation is parameterized in 6D continuous rotation representation

Read the complete text