Linguistics-Vision Monotonic Consistent Network for Sign Language Production

Xu Wang 1 , Shengeng Tang 1∗ , Peipei Song 2 , Shuo Wang 2 , Dan Guo 1 , Richang Hong 1 · 2024

Read Linguistics-Vision Monotonic Consistent Network for Sign Language Production by Xu Wang 1 , Shengeng Tang 1∗ , Peipei Song 2 , Shuo Wang 2 , Dan Guo 1 , Richang Hong 1 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16944.txt.

Introduction

Xu Wang and co-authors build a network that turns sign language glosses into poses while keeping words and movements in step, in a paper from 2024. Making sign videos from spoken sentences hinges on that gloss-to-pose step. The two streams drift apart easily, since no labels mark which gloss matches which motion.

Their transformer design aligns the streams at two scales. A fine-grained aligner keeps the order of glosses and actions consistent by comparing feature sequences. A coarse comparator pulls matched text and video pairs together in meaning and pushes unmatched pairs apart. On the standard PHOENIX14T benchmark, the network beat the prior best systems. The authors argue that both scales of consistency are needed for believable signing.

Catalog record

CREATORXu Wang 1 , Shengeng Tang 1∗ , Peipei Song 2 , Shuo Wang 2 , Dan Guo 1 , Richang Hong 1
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-16944

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Linguistics-Vision Monotonic Consistent Network for Sign Language Production Thanks: * Corresponding author. Xu Wang 1 , Shengeng Tang 1∗ , Peipei Song 2 , Shuo Wang 2 , Dan Guo 1 , Richang Hong 1 Abstract Sign Language Production (SLP) aims to generate sign videos corresponding to spoken language sentences, where the conversion of sign Glosses to Poses (G2P) is the key step. Due to the cross-modal semantic gap and the lack of word-action correspondence labels for strong supervision alignment, the SLP suffers huge challenges in linguistics-vision consistency. In this work, we propose a Transformer-based Linguistics-Vision Monotonic Consistent Network (LVMCN) for SLP, which constrains fine-grained cross-modal monotonic alignment and coarse-grained multimodal semantic consistency in language-visual cues through Cross-modal Semantic Aligner (CSA) and Multimodal Semantic Comparator (MSC). In the CSA, we constrain the implicit alignment between corresponding gloss and pose sequences by computing the cosine similarity association matrix between cross-modal feature sequences ( i.e. , the order consistency of fine-grained sign glosses and actions). As for MSC, we construct multimodal triplets based on paired and unpaired samples in batch data. By pulling closer the corresponding text-visual pairs and pushing apart the non-corresponding text-visual pairs, we constrain the semantic co-occurrence degree between corresponding gloss and pose sequences ( i.e. , the semantic consistency of coarse-grained textual sentences and sign videos). Extensive experiments on the popular PHOENIX14T benchmark show that the LVMCN outperforms the state-of-the-art. I Introduction Sign language production plays a significant role in the deaf community as a rich form of visual language communication. Early works focus on avatar-based [ 1 ] and statistical machine translation methods [ 2 ] . Recently, influenced by the excellence of deep learning, approaches based on CNN&RNN [ 3 , 4 , 5 , 6 ] , GAN [ 7 , 8 ] and VAE [ 9 , 10 ] have emerged. Nowadays, a new common approach is to employ the Transformer framework to decode pose sequences [ 11 , 12 , 13 , 14 , 15 , 16 , 17 ] . Existing methods suffer huge multi-model semantic gaps due to the lack of robust supervised cross-modal alignment labeling. Although cross-modal semantic consistency has been noted in sign language translation [ 18 , 19 , 20 , 21 ] , it has not been addressed in SLP, which needs to generate complex visual semantics from weak text. Nowadays, in many downstream tasks such as cross-modal retrieval [ 22 , 23 , 24 , 25 , 26 ] , vision-language understanding [ 27 , 28 , 29 , 30 ] , multimodal sentiment analysis [ 31 , 32 , 33 , 34 , 35 ] , researchers tackle the rich multimodal data through the alignment and comparison of cross-modal semantics. Therefore, we propose a Linguistics-Vision Monotonic Consistent Network (LVMCN) for SLP, which constrains sign linguistic-visual cues from both fine-grained cross-modal monotonic alignment and coarse-grained multimodal semantic consistency. As shown in Figure 1 , the LVMCN is designed based on a Transformer framework that consists of a Cross-modal Semantic Aligner and a Multimodal Semantic Comparator. Firstly, the textual and visual semantic features considered as original gloss and pose sequences are extracted from gloss encoder and pose decoder of LVMCN respectively. Next, the semantic features obtained above are simultaneously inputted into the CSA and MSC. We automatically constrain the implicit alignment between the corresponding gloss and pose sequences by calculating the cosine similarity correlation matrices to solve the problem of inconsistent semantics of sign glosses and actions. Meanwhile, we construct multimodal triples from paired and unpaired batch data samples in the MSC. Semantic co-occurrence between gloss and pose sequences is enhanced by aligning corresponding text-visual pairs and separating non-corresponding ones, further improving the consistency of multimodal cues in SLP. Compared with previous approaches, our main contributions are summarised below: (1) The proposed CSA constructs monotonic alignment of fine-grained cross-modal sequences in forward (linguistics-to-vision) and reverse (vision-to-linguistics). (2) The designed MSC constructs multimodal triples according to the sentence-video matching relationships in the sample data to constrain semantic consistency between coarse-grained textual sentences and sign videos within batch data. (3) The constructed framework incorporates alignment loss \mathcal{L}_{ali} and comparison loss \mathcal{L}_{com} , in conjunction with classical fitting loss \mathcal{L}_{acc} , to jointly optimize the SLP task. Fig. 1: Overview of LVMCN framework. It contains two key modules: Cross-modal Semantic Aligner (CSA) and Multimodal Semantic Comparator (MSC). II METHOD As illustrated in Figure 1 , we proposed LVMCN is constructed based on a Transformer framework (see Section II-A ), which includes two key modules: Cross-modal Semantic Aligner (CSA) (see Section II-B ) and Multimodal Semantic Comparator (MSC) (see Section II-C ). II-A Transformer-based SLP Framework To explore gloss semantic features, we build a transformer-based encoder. We apply a linear embedding layer to map glosses into a high-dimensional space and a positional coding layer to capture temporal order, formulated as: x^{\prime}_{n}=W^{g}\cdot x_{n}+b^{g}+PE(n), (1) where x_{n} is a one-hot vector of the n -th gloss over the gloss vocabulary \mathcal{V} , PE is conducted by the sine and cosine functions on the temporal gloss order as in [ 12 ] , and W^{g} and b^{g} represent the weight and bias respectively. Then, we use Multi-Head Attention (MHA), Normalisation Layer (NL) and Feedforward Layer (FL) to capture the global semantics of the glosses, i.e. , \tilde{x}^{\prime}_{1:N}=NL(FL(MHA(x^{\prime}_{1:N})) . Similar to the gloss encoding, we encode the sign poses into a high-dimensional feature space through a linear layer and a positional encoding layer PE^{\prime} , formulated as: y^{\prime}_{m}=W^{p}\cdot{y}_{m}+b^{p}+PE^{\prime}(m), (2) where y_{m} denotes the coordinates of the pose at m -th time stamp; W^{p} and b^{p} represent the weight and bias respectively. Our pose decoder is recursive that predicts the next pose \tilde{y}_{1:m} by aggregating all previously generated poses \tilde{y}_{m+1} . This process is formulated as follows: \displaystyle\tilde{y}_{m+1}=PoseDecoder(y^{\prime}_{1:m},\tilde{x}_{1:N}) (3) \displaystyle\Leftrightarrow\left\{\begin{array}[]{l}z_{m}=FL(MHA_{1}(y^{\prime}_{1:m})+\tilde{y}_{m}),m\in[1,M];\\ \tilde{y}_{m+1}=FL(MHA_{2}(z_{m},\tilde{x}_{1:N})),\\ \end{array}\right. where MHA_{1} and MHA_{2} are the two layers of pose decoder. Pose Optimization : In the training stage, the Mean Absolute Error (MAE) loss is used to constraint the consistency of the produced poses \{\tilde{y}\}_{m=1}^{M} and the ground truth \{\widehat{y}\}_{m=1}^{M} . \mathcal{L}_{acc}=\frac{1}{M}\sum_{m=1}^{M}|\tilde{y}_{m}-\widehat{y}_{m}|. (4) II-B Cross-modal Semantic Aligner (CSA) To achieve cross-modal monotonic matching between gloss and pose sequences, we design a fine-grained semantic aligner. Specifically, we automate the constraints on the implicit alignment between the corresponding gloss and pose sequences by computing the cosine similarity correlation matrices between the textual features \{\tilde{x}\}_{n=1}^{N} and the visual features \{z\}_{m=1}^{M} to achieve accurate fine-grained cross-modal sequence monotonic alignment. In practice, we first normalize the textual features \{\tilde{x}\}_{n=1}^{N} and visual features \{z\}_{m=1}^{M} , formulated as: t_{n}=Normalize(\tilde{x}_{n});v_{m}=Normalize(z_{m}). (5) We compute the match between t_{n} and v_{t} by cosine similarity and propose a textual visual semantic best matching function \mathcal{H}(V,T) , which finds the closest item from sequence V=\{v_{1},v_{2},\cdots,v_{M}\} for each item in sequence T=\{t_{1},t_{2},\cdots,t_{N}\} . The calculation process can be expressed as: \displaystyle\left\{\begin{array}[]{l}\mathcal{H}(V,T)_{m}=argmax(h(v_{m},t_{n}));\\ \phi(V,T)=\frac{1}{M}\displaystyle\sum_{m=1}^{M}h(v_{m},\mathcal{H}(V,T)_{m}),\end{array}\right. (6) where h (·, ·) denotes cosine similarity. The result of \phi(V,T) is the video-text similarity, which finds the most similar textual feature for each visual feature. We also calculate the text-video similarity \phi(T,V) with the function \mathcal{H}(T,V) . Alignment Optimization : For a batch containing B text-video pairs \{V_{i},T_{i}\}_{i=1}^{B} , the alignment loss is defined as: \begin{split}\mathcal{L}_{ali}=&-\frac{1}{B}\displaystyle\sum_{i=1}^{B}(log\frac{\exp(\phi({V}_{i},{T}_{i})/\tau)}{\textstyle\sum_{j=1}^{B}\exp(\phi({V}_{i},{T}_{j})/\tau)}\\ &+log\frac{\exp(\phi({T}_{i},{V}_{i})/\tau)}{\textstyle\sum_{j=1}^{B}\exp(\phi({T}_{i},{V}_{j})/\tau)})\end{split} (7) where \phi({V}_{i},{T}_{j}) denotes the similarity

Read the complete text