Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization

Cong Wang 1, Zexuan Deng 1, Zhiwei Jiang 1, Fei Shen 2, Yafeng Yin 1, Shiwei Gan 1 Zifeng Cheng 1 Shiping Ge 1 Qing Gu 1 · 2025

Read Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization by Cong Wang 1, Zexuan Deng 1, Zhiwei Jiang 1, Fei Shen 2, Yafeng Yin 1, Shiwei Gan 1 Zifeng Cheng 1 Shiping Ge 1 Qing Gu 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15980.txt.

Introduction

Cong Wang and co-authors generate sign language videos from spoken text with a richer set of motion cues, in a paper from 2025. Older systems lean on one coarse cue, a skeleton sequence, to bridge text and video. That limit flattens how natural and expressive the signing looks.

Their frame, SignViP, adds fine cues such as detailed poses and 3D hands. A diffusion model learns embeddings that hold both motion and look. An autoencoder then squeezes those embeddings into discrete tokens. A translator turns spoken text into those tokens, which are decoded back to guide the video model. Scores across standard metrics put the system at the top of the field.

Catalog record

CREATORCong Wang 1, Zexuan Deng 1, Zhiwei Jiang 1, Fei Shen 2, Yafeng Yin 1, Shiwei Gan 1 Zifeng Cheng 1 Shiping Ge 1 Qing Gu 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15980

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization Cong Wang 1, Zexuan Deng 1, Zhiwei Jiang 1, Fei Shen 2, Yafeng Yin 1, Shiwei Gan 1 Zifeng Cheng 1 Shiping Ge 1 Qing Gu 1 Abstract Sign Language Video Generation (SLVG) seeks to generate identity-preserving sign language videos from spoken language texts. Existing methods primarily rely on the single coarse condition ( e.g. , skeleton sequences) as the intermediary to bridge the translation model and the video generation model, which limits both the naturalness and expressiveness of the generated videos. To overcome these limitations, we propose SignViP, a novel SLVG framework that incorporates multiple fine-grained conditions for improved generation fidelity. Rather than directly translating error-prone high-dimensional conditions, SignViP adopts a discrete tokenization paradigm to integrate and represent fine-grained conditions ( i.e. , fine-grained poses and 3D hands). SignViP contains three core components. (1) Sign Video Diffusion Model is jointly trained with a multi-condition encoder to learn continuous embeddings that encapsulate fine-grained motion and appearance. (2) Finite Scalar Quantization (FSQ) Autoencoder is further trained to compress and quantize these embeddings into discrete tokens for compact representation of the conditions. (3) Multi-Condition Token Translator is trained to translate spoken language text to discrete multi-condition tokens. During inference, Multi-Condition Token Translator first translates the spoken language text into discrete multi-condition tokens. These tokens are then decoded to continuous embeddings by FSQ Autoencoder, which are subsequently injected into Sign Video Diffusion Model to guide video generation. Experimental results show that SignViP achieves state-of-the-art performance across metrics, including video quality, temporal coherence, and semantic fidelity. The code is available at https://github.com/umnooob/signvip/ . 1 Introduction Sign language, as a visual language, serves as the primary communication medium for deaf individuals. Early research focused on Sign Language Recognition (SLR) [ 27 , 39 , 75 ] , Translation (SLT) [ 3 , 17 , 36 , 12 ] , or Production (SLP) [ 49 , 47 , 51 , 71 , 68 ] . More recently, Sign Language Video Generation (SLVG) [ 48 , 52 , 41 ] has gained increasing attention, which aims to generate realistic and expressive sign language videos from spoken language texts, preserving the unique identity of a target signer, as shown in Figure 1 (1). This growing interest is driven by its potential applications in accessibility technologies, educational tools, immersive communication systems, etc. SLVG presents a challenging problem due to the lack of explicit spatial or temporal alignment between the input ( i.e. , the spoken language texts) and output ( i.e. , the sign language videos) modalities. To address this, current SVLG methods focus on leveraging synchronized auxiliary conditions as an intermediary to align these two modalities. As shown in Figure 1 (2a), most of the existing SLVG methods [ 48 , 52 ] leverage skeletal sequences as an intermediary to bridge a text-to-skeleton translation model ( i.e. , an SLP model) and a skeleton-to-video generation model. Because the generative model is only guided by a single coarse condition ( e.g. , skeleton), such single-conditional methods struggle with the naturalness and expressiveness of the generated videos, particularly in capturing facial expressions and figure movements, as illustrated in Figure 1 (3). Recent advancements in human video generation have shown that incorporating fine-grained conditions ( e.g. , dense pose [ 69 ] or 3D models [ 7 , 67 ] ) or leveraging multiple conditions ( e.g. , depth combined with optical flow [ 70 ] ) can substantially improve generative fidelity. Inspired by these, we consider whether multiple fine-grained conditions can be introduced to enhance the quality and expressiveness of generated sign language videos. As shown in Figure 1 (2b), one intuitive solution is to extend the single-conditional methods by considering multiple fine-grained conditions as intermediaries, where a multi-condition translation model can directly predict capable of achieving multiple fine-grained conditions. However, as shown in Figure 1 (4), we observe that directly translating such attributes is challenging due to their high-dimensional nature and susceptibility to errors. This raises an important question: How can we overcome the challenges in the translation of multiple fine-grained conditions to further advance SLVG? Figure 1: (1) The illustration of SLVG task. (2) The pipeline comparison between existing SLVG methods ( i.e. , single-condition method and multi-condition method) and our SignViP. (3) Single-condition methods struggle to accurately capture the naturalness and expressiveness of sign language videos. (4) Multi-condition methods are prone to translation errors for fine-grained conditions. To address the challenges, we propose SignViP , a novel framework designed to advance SLVG by incorporating multiple fine-grained conditions for enhanced generation fidelity. As shown in Figure 1 (2c), instead of directly translating high-dimensional conditions from spoken language texts, SignViP adopts a discrete tokenization paradigm to effectively integrate and represent these fine-grained conditions. Central to this framework is the construction of a discrete multi-condition token space , which bridges fine-grained conditions ( e.g. , fine-grained poses and 3D hands) with the dynamics of sign language video frames. The framework consists of three key components: (1) Sign Video Diffusion Model is jointly trained with a multi-condition encoder using denoising loss to generate continuous embeddings that encapsulate fine-grained motion and appearance details; (2) Finite Scalar Quantization (FSQ) Autoencoder is trained with reconstruction loss to compress and quantize the continuous embeddings into discrete tokens, enabling highly dense representation for the conditions; (3) Multi-Condition Token Translator is built upon an autoregressive model to translate spoken language text to discrete multi-condition tokens. During inference, the spoken language text is first translated into discrete multi-condition tokens by Multi-Condition Token Translator. These tokens are then decoded back into continuous embeddings by FSQ Autoencoder, which are subsequently injected into Sign Video Diffusion Model to guide sign language video generation. Our experimental results demonstrate that SignViP achieves state-of-the-art performance across multiple evaluation metrics, including video quality, temporal coherence, and semantic fidelity. Our main contributions are summarized as follows: • We introduce SignViP, a novel framework for Sign Language Video Generation (SLVG) that incorporates multiple fine-grained conditions for improved video quality and expressiveness. • We propose a discrete tokenization paradigm through the construction of a discrete multi-condition token space to bridges fine-grained conditions with the dynamics of sign language video frames. • The experiments validate the effectiveness of SignViP, demonstrating state-of-the-art performance across diverse metrics. 2 Related Works Sign Language Video Generation. Sign Language Video Generation (SLVG) aims to generate identity-preserving sign language videos from spoken language texts. Early methods decompose the task into two consecutive sub-tasks [ 48 , 52 ] , which are the text-to-skeleton translation ( i.e. , SLP) and the skeleton-to-video generation. SignGAN [ 48 ] first employs a transformer [ 61 ] with a mixture density formulation to translate spoken language text to skeletal sequence. Then, a GAN-based [ 15 ] skeleton-conditioned human synthesis model is introduced to generate sign language videos. FS-Net [ 52 ] extends SignGAN by predicting the temporal alignment to a continuous signing sequence. Because the single condition focuses solely on capturing basic pose structures while neglecting fine-grained details, the generated videos tend to appear less natural and expressive. SignGen [ 41 ] seeks to address this limitation through a novel end-to-end pipeline that integrates multi-condition guidance, including optical

Read the complete text