Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation

Yiyang Jiang, Li Zhang, Xiaoyong Wei, Qing Li · 2026

Read Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation by Yiyang Jiang, Li Zhang, Xiaoyong Wei, Qing Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15301.txt.

Introduction

Jiang and three colleagues rethink gloss-free sign language translation, which converts signing video to text without an intermediate sign glossary. Signers build meaning on the fly from context, space, and movement, so the team inserts an ordered chain of latent thoughts between video and text. They then plan wording before grounding it in video evidence. Tests across benchmarks beat prior gloss-free methods, aided by a new large dataset.

Catalog record

CREATORYiyang Jiang, Li Zhang, Xiaoyong Wei, Qing Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67780
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15301

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation Yiyang Jiang, Li Zhang, Xiaoyong Wei, Qing Li Abstract Many Sign language translation (SLT) systems quietly assume that brief chunks of signing map directly to spoken-language words. That assumption breaks down because signers often create meaning on the fly using context, space, and movement. We revisit SLT and argue that it is mainly a cross-modal reasoning task, not just a straightforward video-to-text conversion. We thus introduce a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit middle layer between the video and the generated text. These latent thoughts gradually extract and organize meaning over time. On top of this, we use a plan-then-ground decoding method: the model first decides what it wants to say, and then looks back at the video to find the evidence. This separation improves coherence and faithfulness. We also built and released a new large-scale gloss-free SLT dataset with stronger context dependencies and more realistic meanings. Experiments across several benchmarks show consistent gains over existing gloss-free methods. Our code and data are available at https://github.com/fletcherjiang/SignThought . 1 Introduction Sign language translation (SLT) is critically important both as a vital assistive technology for connecting Deaf and hard-of-hearing communities and as a challenging multimodal task within natural language processing Bragg et al. 2019 . Research in this area has evolved significantly, progressing from early methods that approached SLT as a gloss-level classification problem over video segments Koller et al. 2015 to more recent formulations as a gloss-free, video-to-text sequence transduction task Lin et al. 2023 ; Chen et al. 2024a , tackled by specialized architectures or multimodal large language models Wong et al. 2024 ; Chen et al. 2024b ; Gong et al. 2024 . While these efforts have yielded encouraging outcomes, substantial room for improvement remains. Figure 1: Latent thoughts clarified the context to disambiguate signs and guide decoding toward the correct object interaction in gloss-free SLT. A key limitation lies in a long-standing misconception in model design: treating sign video segments as if they were directly mappable to static spoken-language words or phrases Yin et al. 2021 ; Camgoz et al. 2018 ; Bragg et al. 2019 . This view originates from an early emphasis on the Frozen Lexicon , i.e., the finite inventory of conventionalized signs listed in glossaries, while largely overlooking Productive Forms Forster et al. 2014 . Productive forms are constructed on-the-fly through classifiers, spatial grammar, and motion modulation, forming an open-ended and context-governed system that cannot be enumerated as discrete lexical entries Zwitserlood 2012 . For instance, a single “vehicle” handshape may express meanings such as “park,” “crash,” or “drive” purely through variations in movement and spatial configuration Zwitserlood 2012 ; Ngo et al. 2008 . As a consequence, meaning in sign language is often not explicitly encoded in fixed symbolic units but dynamically generated from motion, space, and discourse context Zwitserlood 2012 ; Cormier et al. 2012 ; Xiao-Yong Wei 2013 . SLT is therefore a generative, context-dependent reasoning problem, not just segment alignment or symbol substitution Yin et al. 2021 ; Bragg et al. 2019 ; Gong et al. 2024 ; Jiang et al. 2024 . From this view, SLT naturally aligns with recent advances in text reasoning techniques, where models explicitly maintain intermediate semantic states to support multi-step meaning composition under weak or implicit supervision Wei et al. 2022 ; Yao et al. 2022 . However, unlike text-only reasoning, SLT reasoning must operate across heterogeneous visual and linguistic modalities Mitra et al. 2024 . The absence of discrete reasoning primitives and explicit intermediate representations makes existing reasoning techniques difficult to transfer directly, leaving multimodal reasoning in SLT largely unexplored Yin et al. 2021 . We therefore position this work as a pilot study toward understanding how explicit reasoning can be modeled in gloss-free SLT, with the goal of identifying fundamental challenges and exploring principled solutions Bragg et al. 2019 ; Jiang et al. 2026b . This paradigm offers three key advantages that tackle the reasoning challenges as: • Latent Thought Abstraction. We introduce latent thoughts as an explicit intermediate semantic interface between sign videos and text generation. Rather than compressing all semantic information into opaque encoder features, the model maintains an ordered set of latent reasoning states that progressively distill and organize meaning from long, continuous visual streams, thereby bridging the mismatch between continuous visual evidence and discrete reasoning primitives. • Plan–Ground Decoupling. Reasoning and grounding are explicitly separated through a plan-then-ground generation mechanism: the model first determines what semantic content to express by reasoning over latent thoughts, and only then decides where to retrieve the corresponding visual evidence. This decoupling alleviates the strong entanglement between semantic decision-making and evidence retrieval in existing SLT systems, improving controllability and grounding faithfulness. • Traceable Evidence Alignment. Latent thoughts function not only as internal planning states but also as traceable anchors that align generated text with specific temporal regions of the input video, enabling explicit evidence attribution and more faithful translations, while a newly constructed large-scale gloss-free SLT dataset provides the essential empirical basis for studying and validating such reasoning behaviors. 2 Related Work Sign Language Recognition and Translation. Sign Language Recognition (SLR) maps sign videos to gloss sequences, evolving from isolated (ISLR) Albanie et al. 2020 ; Li et al. 2020a to continuous settings (CSLR), with typical pipelines extracting visual features from RGB or pose-based inputs Chen et al. 2022b , modeling temporal dynamics via RNN/LSTM or Transformer architectures Camgoz et al. 2018 ; Camgoz et al. 2020 , and decoding glosses using HMMs Koller et al. 2017 or CTC objectives Cheng et al. 2020 . While effective, such systems are computationally intensive and limited in exploiting higher-level linguistic context. Sign Language Translation (SLT) further translates sign videos into spoken or written language, where most methods remain gloss-based by cascading SLR with translation or jointly optimizing both tasks Camgoz et al. 2020 ; Zhou et al. 2021 . In contrast, gloss-free SLT directly learns video-to-text mappings using Transformer-based or variational models Li et al. 2020b ; Camgoz et al. 2018 ; Tu and Weng 2026 ; although recent advances leveraging large-scale multimodal pretraining Li et al. 2025 and large language model adaptation Wong et al. 2024 improve translation fluency, they introduce substantial computational overhead and increased dependence on external corpora. Figure 2: An overview of the SignThought framework, which consists of three parts: (i) a sign encoder that maps the input video \mathbf{X} into dense evidence features \mathbf{E} ; (ii) a Latent Chain-of-Thought module that iteratively updates K learnable thought slots and performs structured routing (segmentation \rightarrow Sinkhorn-style binding \rightarrow routed retrieval, with an optional monotonic bias) to distill \mathbf{E} into an ordered thought chain \mathbf{C} ; and (iii) a plan-then-ground dual-stream decoder that first attends to \mathbf{C} for semantic planning and then grounds each token on \mathbf{E} under thought-guided temporal priors to generate the spoken-language translation end-to-end. Reasoning and Latent Thoughts. Chain-of-Thought (CoT) prompting enhances multi-step reasoning in large language models (LLMs) by eliciting intermediate reasoning steps Wei et al. 2023 ; Zhou et al. 2024b ; Zhou et al. 2025 , with extensions such as self-consistency Wang et al. 2023 , planning,

Read the complete text