Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation
Md. Tariquzzaman, Md Farhan Ishmam, Saiyma Sittul Muna, Md. Kamrul Hasan, Hasan Mahmud · 2025
Read Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation by Md. Tariquzzaman, Md Farhan Ishmam, Saiyma Sittul Muna, Md. Kamrul Hasan, Hasan Mahmud (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16076.txt.
Introduction
Tariquzzaman and co-authors teach vision-language models to explain Bengali sign language, gesture by gesture, in a 2025 paper. Step-by-step written guides let hearing people copy signs and truly talk back. The team built BdSLIG, the first dataset for this task in Bengali sign language, a language the big models almost surely never saw in training.
Their prompting trick folds the classic parts of a sign, hand shape, motion, and orientation, straight into the prompt. The guides came out more structured and easier to copy than free-form answers. The work aims at learning tools for the many sign languages that AI has so far skipped.
Catalog record
| CREATOR | Md. Tariquzzaman, Md Farhan Ishmam, Saiyma Sittul Muna, Md. Kamrul Hasan, Hasan Mahmud |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16076
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation Md. Tariquzzaman, Md Farhan Ishmam, Saiyma Sittul Muna, Md. Kamrul Hasan, Hasan Mahmud Abstract Sign Language (SL) enables two-way communication for the deaf and hard-of-hearing community, yet many sign languages remain under-resourced in the AI space. Sign Language Instruction Generation (SLIG) produces step-by-step textual instructions that enable non-SL users to imitate and learn SL gestures, promoting two-way interaction. We introduce BdSLIG, the first Bengali SLIG dataset, used to evaluate Vision Language Models (VLMs) (i) on under-resourced SLIG tasks, and (ii) on long-tail visual concepts, as Bengali SL is unlikely to appear in the VLM pre-training data. To enhance zero-shot performance, we introduce Sign Parameter-Infused (SPI) prompting, which integrates standard SL parameters, like hand shape, motion, and orientation, directly into the textual prompts. Subsuming standard sign parameters into the prompt makes the instructions more structured and reproducible than free-form natural text from vanilla prompting. We envision that our work would promote inclusivity and advancement in SL learning systems for the under-resourced communities. 1 Introduction Sign Language (SL) differs fundamentally from spoken or written languages as it inherently requires communication in the visual modality. At a lexical level, these SL gestures are typically encapsulated in a short video of a few seconds [ 1 , 2 ] . With the current advancements in computer vision models [ 3 , 4 , 5 ] , there has been significant work on automated processing of SL video clips, primarily for American Sign Language (ASL) recognition tasks [ 6 , 7 ] . Despite the proliferation of ASL recognition works, we highlight two major research gaps in this domain. Firstly, the overwhelming focus on ASL works has marginalized non-ASL languages [ 8 ] , limiting the inclusivity and generalizability of the models [ 9 ] . Secondly, SL recognition addresses only one half of the SL communication process, i.e . , understanding the SL user. The other half, i.e . , communicating back to the SL user, requires learning and producing the SL gestures - a task which is relatively less popular. Current literature has provided methods for non-ASL and under-resourced SL recognition tasks [ 8 , 10 ] . For learning SL, LLM-based automated SL Instruction Generation (SLIG) [ 11 ] creates textual instructions that guide the users to replicate the SL gesture. Automated gesture synthesis [ 12 ] can also be a plausible option for communicating back to the SL user, albeit with limited real-life applicability. However, there has been no prior work on the intersection of the aforementioned research problems, i.e . , SLIG for low-resource languages. Our work serves as a pilot project to address this problem space. SLIG in non-ASL languages is critical, as each SL possesses unique gestural cues, shaped by linguistic, cultural, and regional factors [ 13 , 14 ] . These variations mirror linguistic differences in spoken or written languages. We chose Bengali for our non-ASL evaluation language as it remains an under-resourced language in the automated SL space [ 15 ] , while being native to 300 million active speakers, making it a high-impact choice for real-world applicability. Bengali SL also predominantly focuses on recognition tasks [ 2 , 16 , 17 ] ; hence, our proposed BdSLIG is the first low-resource and Bengali SLIG dataset. We hypothesize that a Vision Language Model (VLM) [ 18 , 19 ] , when given a frame or short video of a sign, can generate a textual description ( e.g. , “both hands raised near face, wrists bent inwards”) that captures the sign’s meaning and form. Such descriptions could serve as interactive instructions for learners (analogous to language learning feedback [ 20 ] ) and also provide interpretability by explaining what the model “sees.” This idea builds on prior work in explainable sign technology, e.g . , the Learn2Sign system [ 21 ] , which provided feedback on location, movement, and handshape to ASL learners. Since Bengali Sign Language is a low-resource language and sign recognition models are still underdeveloped, it is essential to analyze how large models behave in this context. Prior research [ 22 ] has shown that general LLMs may produce overly generic or semantically off-base sign descriptions without domain-specific guidance. In light of these limitations, our contributions can be summarized as follows: • We propose the first dataset to evaluate the performance of VLMs on the under-resourced task of Bengali sign language instruction generation. • We introduce Sign Parameter-Infused (SPI) prompting, which integrates standard sign language parameters, e.g . , hand shape, motion, and orientation, directly into the textual prompts of the VLMs. Experiments reveal SPI prompting generally leads to better performance. • Our experiments also serve as an evaluation of the performance of VLMs on long-tail distributed and semantically fine-grained visual concepts. 2 Methodology In this section, we first define the sign language parameters (§ 2.1 ) that will be relevant to our prompting technique (§ 2.3 ). We also construct the BdSLIG dataset (§ 2.2 ) and define our evaluation framework (§ 2.4 ). 2.1 Sign Language Parameters Sign language parameters are essential visual cues used to identify or describe a sign language gesture [ 23 ] . Our work uses the following parameter categories, defined below with a few associated keyword tags: 1. Handshape : The morphological configuration, e.g . , raised, extended, pointed, bent, spread, twisted, of the fingers and palms. 2. Movement Type : The trajectory or action of the hands, e.g . , static, circular, squeezing, grabbing, rotating, pulling. 3. Location : The placement of the hands while producing the sign, e.g . , near head, forehead, eye, ear, nose, mouth, chest, shoulder. 4. Palm Orientation : The direction the palm is facing relative to the signer’s body, e.g . , inward, outward, upward, downward, left, right. 5. Spatial Interaction : The role of one or both hands in the sign’s spatial scheme, e.g . , right-hand active, left-hand active, both-hands symmetric, both-hands asymmetric. 6. Temporal Dynamics : The temporal pattern of the movement onset, duration, and repetition, e.g . , sustained hold, repeated oscillation, single tap, repeated tapping. 7. Facial Cues : Facial expressions, e.g . , neutral, brows-raised, brows-furrowed, mouth-open, mouth-closed, lip-pursed, that can disambiguate the sign language. These sign-language parameters serve two roles in our work: (i) to establish the annotation guidelines for human annotation of the BdSLIG dataset, and (ii) to define the Sign Parameter-Infused (SPI) prompting for evaluating zero-shot capabilities of vision-language models (VLMs). Fig. 1 provides a visual overview of these parameters, where each category is illustrated with example video frames sampled from the BdSLIG dataset. Figure 1 : Parameter-based description of the Toothpaste sign (video taken from the BdSLIG dataset). The numbers denote the parameter numbers mentioned in § 2.1 . The parameter descriptions are given to elucidate and exemplify the given SL parameters and are not part of the BdSLIG dataset. 2.2 BdSLIG Dataset The images of BdSLIG are sourced from the BdSLW60 dataset [ 15 ] , a 60-word Bengali sign language recognition dataset. For each word, we selected one representative video and manually annotated the textual sign instructions. We hired three trained annotators following the annotation guidelines with our