Real Time Captioning of Sign Language Gestures in Video Meetings

Sharanya Mukherjee, Md Hishaam Akhtar, Dr Kannadasan R · 2025

Read Real Time Captioning of Sign Language Gestures in Video Meetings by Sharanya Mukherjee, Md Hishaam Akhtar, Dr Kannadasan R (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14543.txt.

Introduction

Sharanya Mukherjee, Md Hishaam Akhtar, and Kannadasan propose live captions for sign language inside video meetings. Deaf signers often prefer signing to typing on calls, yet few hearing colleagues know sign. Computer vision that turns signs into subtitles could close that gap.

The team builds a browser extension that watches the camera feed and posts translated captions for everyone on the call. Training leans on a large public data set of word-level American Sign Language clips, over two thousand videos from more than one hundred signers. The paper frames the tool as an accessibility layer for the meeting software people already use daily.

Catalog record

CREATORSharanya Mukherjee, Md Hishaam Akhtar, Dr Kannadasan R
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-14543

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real Time Captioning of Sign Language Gestures in Video Meetings Sharanya Mukherjee, Md Hishaam Akhtar, Dr Kannadasan R Abstract It has always been a rather tough task to communicate with someone possessing a hearing impairment. One of the most tested ways to establish such a communication is through the use of sign based languages. However, not many people are aware of the smaller intricacies involved with sign language. Sign language recognition using computer vision aims at eliminating the communication barrier between deaf-mute and ordinary people so that they can properly communicate with others. Recently the pandemic has left the whole world shaken up and has transformed the way we communicate. Video meetings have become essential for everyone, even people with a hearing disability. In recent studies, it has been found that people with hearing disabilities prefer to sign over typing during these video calls. In this paper, we are proposing a browser extension that will automatically translate sign language to subtitles for everyone else in the video call. The Large-scale dataset which contains more than 2000 Word-Level ASL videos [ 12 ] , which were performed by over 100 signers will be used. 1 Introduction The use of sign based linguistics have become increasingly popular among humans with a hearing impairment. However, currently, there are only around 250,000 to 500,000 people using American Sign language which significantly limits the number of people with whom they can communicate with ease. The only substitute for sign language is through writing, which is quite cumbersome, impersonal, and impractical in real- life situations, especially during video meetings in this pandemic. To overcome this problem and to enable dynamic communication during video calls, we present a sign language recognition system that will automatically translate sign language to text in real-time and display it as subtitles for everyone in the video call. Sign language recognition where individual words will be identified at a time, and sign language recognition where individual characters will be identified at a time in real time. In this paper, we target word level American sign language identification in real- time using deep learning techniques. Our work will broadly consist of three tasks to be done in real-time: • Video acquisition: Obtaining the video of the person signing • Frame analysis and classification: Analyzing and classifying the frames in the video • Prediction display: Displaying the most likely predicted word From the perspective of computer vision, this work is significantly challenging due to various considerations including environmental concerns like lighting or camera position or background, occlusion like a part of hand being out of the range of vision of the camera, detecting the boundary between each signs (Detecting when a sign ends and the next begins) Previously neural networks were used to recognize character-level American sign language with accuracies consistently over 90 percent, but most of them require an element for 3-D capture through motion-tracking gloves or a similar costly hardware devices , and only very few of them provide real-time classifications. These restrictions imposed by hardware reduce the feasilbility and scalibility of these solutions. Our proposed system features a pipeline that takes video of the user who is signing through their local webcam. Next, individual frames are extracted from the video, and prediction is made on it by our model. The predicted word or character will be then broadcast over a socket connection into a room which the participants in the video meeting will be joining through our browser extension. The translated word or character is then displayed, as a subtitle for every user present in the meeting. This system enables dynamic real-time and hassle-free communication between hearing disabled people and others. To make a practical American Sign Language or ASL identification model, the training data needs to contain a massive amount of classes and training examples. For word- level American Sign Language recognition, we will be using the WLASL dataset [ 12 ] , which contains a large-scale lexicon of signs and their corresponding annotations. The videos in the dataset are RGB-based videos that are scraped from the Internet, since we don’t want to anchorage the minimum hardware requirement for the sign identification. With this our trained sign language identification models will not have to depend on unique additional requirements such as Microsoft Kinect or special gloves. 2 Related Work Author Siming [ 1 ] developed a new methodology to tackle the recognition of words. He constructed a dataset which contained 40 words which were frequently used in conversation. Also, he collected 10,000 related images. A region based convolutional neural network coupled with an embedded RPN or region proposal network module was used to detect the palms in action from the media. Such a technique bolstered performance of the model. Also, for the extraction of complex features, a three-dimensional CNN was used. Author Rekha J [ 2 ] employed the use of YCbCr colour model to successfully recognize the human skin from the dataset. A principal curvature based region detector was employed for the extraction of minute features from the images. They were classified using standard algorithms like DTW, Multiclass SVM and a non linear KNN. In another paper [ 3 ] , a color based technique was employed for inexpensive processing. The background of the images were set to green to enable the extraction of the skin from its green surroundings. The extracted results were then transformed into its greyscale counterparts. It was successful in mapping the hand gestures and produced a staggering accuracy of 92%. M. Geetha along with U.C. Manjusha employed [ 4 ] the use of individual alphabet and digits to recognize characters from the video. They extracted the skin region and removed its boundary to perform further analysis on it through a B-Spline Curve. Thereafter, a simple SVM was used for categorising the images. It produced a rather excellent accuracy of 90 Author L. Pigou [ 5 ] utilized the CLAP14 [ 6 ] dataset which contained gestures from the Italian language. Pigou [ 5 ] used a Convolutional Neural network after processing the images. The model had six layers for training and all the kernels are in 2D. The activation function which he used in his work is ReLU or Rectified Linear Units. CNN performs feature extraction. The classification was made possible through the employment of an ANN. The accuracy of the model was 91.70% with an error rate of 8.3%. Author J Huang [ 7 ] built a dataset using Microsoft Kinect by using 25 frequently used words. A three-dimensional CNN was used with 5 main input channels. The aver- age accuracy was about 94.2%. In a research work on gesture recognition done by author J.Carriera [ 8 ] , the transfer learning method was used. ImageNet [ 9 ] and Kinect [ 6 ] datasets were used as the pre-trained datasets for his work. The RGB, flow and pre-trained kinetic was merged after training the model on multiple datasets. On the UCF-101 [ 10 ] dataset, he achieved an accuracy of 98% and 80.9% on the HMDB-51 [ 11 ] dataset. 3 Implementation 3.1 Classifier Model To achieve high precision and accuracy and low cost, transfer learning using MobileNetV2 was used. Transfer Learning

Read the complete text