Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation

Jianyuan Guo, Peike Li, Trevor Cohn · 2025

Read Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation by Jianyuan Guo, Peike Li, Trevor Cohn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15438.txt.

Introduction

Translating sign language video into written text often relies on glosses, word labels for signs, but expert gloss labels are costly and rare. Jianyuan Guo, Peike Li, and Trevor Cohn propose making draft glosses automatically instead. They prompt a large language model with a few example pairs to write pseudo glosses from spoken text.

To match the drafts to real signing, they reorder the pseudo glosses using weak supervision from video, then train a vision encoder and translator in three stages. The approach beat earlier gloss-free systems on two benchmarks and rivaled methods that use real glosses. The authors argue the trick keeps the helpful gloss structure without the labeling cost.

Catalog record

CREATORJianyuan Guo, Peike Li, Trevor Cohn
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15438

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation Jianyuan Guo, Peike Li, Trevor Cohn Abstract Sign Language Translation (SLT) aims to map sign language videos to spoken language text. A common approach relies on gloss annotations as an intermediate representation, decomposing SLT into two sub-tasks: video-to-gloss recognition and gloss-to-text translation. While effective, this paradigm depends on expert-annotated gloss labels, which are costly and rarely available in existing datasets, limiting its scalability. To address this challenge, we propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses while preserving the structured intermediate representation. Specifically, we prompt a Large Language Model (LLM) with a few example text-gloss pairs using in-context learning to produce draft sign glosses from spoken language text. To enhance the correspondence between LLM-generated pseudo glosses and the sign sequences in video, we correct the ordering in the pseudo glosses for better alignment via a weakly supervised learning process. This reordering facilitates the incorporation of auxiliary alignment objectives, and allows for the use of efficient supervision via a Connectionist Temporal Classification (CTC) loss. We train our SLT model—consisting of a vision encoder and a translator—through a three-stage pipeline, which progressively narrows the modality gap between sign language and spoken language. Despite its simplicity, our approach outperforms previous state-of-the-art gloss-free frameworks on two SLT benchmarks and achieves competitive results compared to gloss-based methods. 1 Introduction Sign languages are the primary means of communication for the deaf and hard-of-hearing communities smith1993deafness , functioning as independent linguistic systems with unique lexicons, grammars and syntax armstrong1995gesture . Unlike spoken languages, they convey meaning using a combination of manual gestures such as hand shapes, movements, and positioning, and non-manual cues, including facial expression, mouthing and eyebrow movements koller2019weakly . Despite their crucial role in communication, sign languages often lack standardized written forms (so-called glossing , which we discuss shortly), creating significant challenges for both accessibility and automatic translation. Sign Language Translation (SLT) has emerged as a key research area aimed at converting sign language videos into textual representation keti ; camgoz2020sign ; phx ; sltunet ; mmtl . However, for many reasons, ranging from the complexity of the input (video) to their linguistic complexity, SLT remains a challenging problem within the field of computer vision and natural language processing. The task of SLT is typically framed as a sequence-to-sequence problem, taking video input of a signer, which is processed by vision encoder, and then generating a text output in written language (e.g., English) using a translation decoder, such as an LSTM or Transformer. These approaches can be broadly categorized into gloss-based and gloss-free methods, based on whether they incorporate gloss annotations. Glosses are a written representation of sign language, denoting the sequences of signs used, which often differs substantially from the words used in the written language text, in terms of word choice, grammar and ordering, alongside sign-language specific concepts such as pronominal referents, classifiers, and non-manual signs. Overall, as a linguistic description of sign language, glosses should provide a much richer source of supervision for understanding signed video input compared to the use of written language (closed captions). Dataset Gloss Language Duration (h) Vocabulary (K) SOTA BLEU4 train val test total train val test total Gloss-based Gloss-free \Delta Phoenix14T phx ✓ DGS 9.2 0.6 0.7 11 2 0.9 1 2.9 29.0 † guan2024multi 24.3 † hwang2024efficient 4.7 CSL Daily csl ✓ CSL 20.6 1.2 1.4 23 2 1.3 1.3 2.3 25.8 † 2s-slt 20.6 † hwang2024efficient 5.2 BOBSL bob ✓ BSL 1236 20 204 1467 72 14 35 78 7.3 ‡ sltcc 3.3 sltcc 4.0 How2Sign how2sign ✗ ASL 69.6 3.9 5.6 79 15.6 3.2 3.6 16 - 15.5 ssvpslt - OpenASL openasl ✗ ASL - - - 288 - - - 33 - 21.2 vap - Youtube-ASL youtubeasl ✗ ASL - - - 984 - - - 60 - - - Table 1: Comparison of sign language translation datasets. † Results are sourced from Papers With Code as of April 2025 (large-scale pre-training methods unisign are excluded). ‡ Indicates the oracle setting where previous ground truth sentences are used in sltcc . Gloss-based methods typically involve pre-training the visual encoder on Sign Language Recognition (SLR) cooper2011sign ; rastgoo2021sign , converting videos into gloss sequences. The task then becomes a standard machine translation problem, mapping these glosses to spoken language text. However, these methods are limited by their reliance on expert-annotated gloss labels, which are time-consuming and costly to produce, as glossing is not widely used in the deaf community. Accordingly, glosses are scarce in existing datasets how2sign ; youtubeasl ; openasl . In contrast, gloss-free methods mmtl ; csgcr have gained attention for directly training models on sign videos and corresponding spoken language text, bypassing the need for gloss annotations. While these methods streamline the pipeline and reduce annotation burdens, they face a performance gap compared to gloss-based approaches, as shown in Table 1 . This discrepancy arises from the inherent differences between sign and spoken languages, making direct translation more challenging. In this context, gloss annotations remain vital for bridging these linguistic gaps and ensuring effective translation. Recent approaches seek to leverage approximate pseudo glosses to bridge the semantic gap between visual and textual features gfslt ; vap . For example, Sign2GPT sign2gpt synthesizes pseudo glosses by filtering out function words based on their part-of-speech tags, retaining only nouns, adjectives, and numerals. However, real gloss sequences often contain terms not in the original text, and their order often differs from the written text. This misalignment between pseudo-glosses and video inputs complicates the use of conventional training methods, such as Connectionist Temporal Classification (CTC) loss ctc , reducing their effectiveness. To combine the advantages of gloss-free methods with improved performance, we propose a framework that generates pseudo glosses using LLMs. This approach enables us to decompose the SLT process into pre-training and fine-tuning stages, as illustrated in Figure 1 , bridging the gap between visual signals and spoken language, and allowing the SLT model to leverage gloss-like structures for enhanced performance. The core idea is that the semantic information of sign language is inherently embedded in reference translations provided alongside the video in standard SLT datasets, making it possible to systematically derive approximate glosses from it. By harnessing the reasoning capabilities of LLMs gemini ; gpt4 , we extract key information from the text to generate pseudo glosses. To further improve this process, we utilize the in-context learning ability of LLMs, providing a small set of example text-gloss pairs ( e.g. , 30). This allows the model to better capture the underlying structure and generate pseudo-glosses that align with sign representations. To mitigate the potential misalignment between LLM-generated glosses and the actual sign sequences in videos, we introduce a reordering method, framed as a weakly supervised learning problem. We train an order-invariant classifier to predict the set of glosses in each video, which is then used to infer a temporal alignment. Armed with reordered pseudo glosses, we decompose the SLT task into three key stages mmtl : video to pseudo gloss recognition (Sign2Gloss), pseudo gloss to text translation (Gloss2Sign), and final fine-tuning (Sign2Text), as illustrated in Figure 1 . This structured breakdown alleviates the challenges

Read the complete text