Integrating Persian Lip Reading in Surena-V Humanoid Robot for Human-Robot Interaction
Ali Farshian Abbasi, Aghil Yousefi-Koma, Soheil Dehghani Firouzabadi, Parisa Rashidi, Alireza Naeini · 2025
Read Integrating Persian Lip Reading in Surena-V Humanoid Robot for Human-Robot Interaction by Ali Farshian Abbasi, Aghil Yousefi-Koma, Soheil Dehghani Firouzabadi, Parisa Rashidi, Alireza Naeini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13996.txt.
Introduction
Ali Farshian Abbasi and four co-authors report, in a 2025 paper, teaching a humanoid robot to read lips in Persian. Watching lips helps machines grasp speech in noisy rooms and serves roles such as care and customer help. No Persian lip-reading data set existed, so the team built one and fitted the skill into the Surena-V robot.
They tried two routes. One tracks facial landmarks, above all around the lips, and infers the movement. The other feeds raw video to a mix of convolutional and memory networks. The memory-based model scored best, at eighty-nine percent accuracy, and now runs on the robot in real time. The work shows lip reading can widen a robot's grasp of speech where sound is limited.
Catalog record
| CREATOR | Ali Farshian Abbasi, Aghil Yousefi-Koma, Soheil Dehghani Firouzabadi, Parisa Rashidi, Alireza Naeini |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13996
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Integrating Persian Lip Reading in Surena-V Humanoid Robot for Human-Robot Interaction Ali Farshian Abbasi, Aghil Yousefi-Koma, Soheil Dehghani Firouzabadi, Parisa Rashidi, Alireza Naeini Abstract Lip reading is vital for robots in social settings, improving their ability to understand human communication. This skill allows them to communicate more easily in crowded environments, especially in caregiving and customer service roles. Generating a Persian Lip-reading dataset, this study integrates Persian lip-reading technology into the Surena-V humanoid robot to improve its speech recognition capabilities. Two complementary methods are explored, an indirect method using facial landmark tracking and a direct method leveraging convolutional neural networks (CNNs) and long short-term memory (LSTM) networks. The indirect method focuses on tracking key facial landmarks, especially around the lips, to infer movements, while the direct method processes raw video data for action and speech recognition. The best-performing model, LSTM, achieved 89% accuracy and has been successfully implemented into the Surena-V robot for real-time human-robot interaction. The study highlights the effectiveness of these methods, particularly in environments where verbal communication is limited. \ps@headings I Introduction Lip reading, or speech reading, is the skill of understanding spoken language by observing lip, facial, and tongue movements without hearing the words. This complex ability is especially useful in noisy settings or for those with hearing impairments [1]. Lip reading plays a critical role in both human-to-human and human-computer interactions, acting as an essential communication tool for individuals with verbal limitations [2],[3]. Without auditory cues, visual signals become crucial for recognizing speech. However, lip reading has inherent challenges, such as differentiating between homophones [4] and handling variations in lip movements for the same phrases among different speakers. Additionally, the orientation of a person’s face can significantly affect recognition [5]. Humanoid robots, created to replicate human form and behaviors, are increasingly prevalent across various industries. These robots are designed to take on tasks traditionally performed by humans [6]. With ongoing technological advancements, humanoid robots are proving valuable in settings where human-like interaction is essential. By integrating lip-reading recognition technology, these robots can interpret visual cues when verbal communication is obstructed, such as at busy events or in noisy industrial spaces. This function supports smooth interaction and enhances efficiency in crowded environments [7]. Fig. 1 : The location of marks and their index in real and 3D model picture Considerable research has been dedicated to advancing lip-reading technology, leading to various datasets and methodologies. For example, Javad Peymanfard et al. [8] created a Persian word dataset of 244,000 videos featuring celebrities from the Persian platform Aparat. This study used the AV-Hubert model, achieving 21.43% accuracy. Similarly, Mahsa Hedayatipour et al. [9] developed a Persian dataset with 30 hours of video from 40 speakers, using a Hidden Markov Model (HMM) to classify six Persian words with 47.73% accuracy. Sivri created a Turkish dataset, containing three phrases and three words sourced from Turkish YouTube videos [10]. The dataset was evaluated using Convolutional Neural Networks (CNN), Long Short-Term Memory Networks (LSTM), and Bidirectional Gated Recurrent Units (BGRU). Aripin et al. [11] compiled an Indonesian lip-reading dataset containing ten words and four phrases spoken by ten participants, reaching 95% accuracy with Long Short-Term Memory Networks (LRCN). Faisal et al. [12] built an Urdu dataset featuring ten participants who repeated ten words and phrases ten times each. Although phrase recognition accuracy was challenging, the LSTM model achieved 62%. Dweik produced an Arabic dataset of 1,051 videos with male and female speakers enunciating common words under various lighting and angle conditions, achieving up to 82.86% accuracy using three different deep-learning models [13]. This study gathered a comprehensive dataset at the Centre for Advanced Science and Technology (CAST), recorded by laboratory research students. Various classification techniques were tested for their ability to recognize lip movements, identifying the most accurate method, which was subsequently applied to the Surena V humanoid robot. This technology allows the robot to recognize words and respond accordingly with gestures, facial expressions, or vocalizations that match the intended message. This work demonstrates the potential of advanced lip-reading technology to improve human-robot interactions in environments where verbal communication is limited. The following sections introduce the dataset, explain the methodology, and detail the approaches tested. The next step involves implementing the best method on the Surena V robot. Finally, the study’s results and their implications for human-robot interaction are discussed. Dataset For Persian, the lipreading model has been trained to recognize seven common words, as shown in Table I, along with their English pronunciations and meanings. These words were selected based on criteria such as usage frequency, simplicity, coherence, and their applicability for commanding the Surena humanoid robot. Data for this project was collected using the robot’s frontal RGB-D camera, recording videos at 20 fps to enhance model accuracy. An RGB-D camera captures both color RGB and depth D information in real-time, combining traditional 2D image data with depth perception using infrared sensors. This enables it to create 3D representations of objects and environments, making it ideal for applications in robotics, AR/VR, and 3D scanning[14] The study involved 20 participants (16 men and 4 women), each pronouncing the selected words 20 times with a 1-second interval between repetitions. The original videos were segmented so that each clip contained a single pronounced word, resulting in 20 smaller videos per recording. Initially, the videos had a resolution of 1280×720, which was suboptimal for tracking lip movement. To improve focus on the lips, the Cvzone face detector was employed, reducing the video size to 300×300. After these adjustments, the total number of video clips increased to 2,800. Unlike other datasets, this dataset emphasizes facial features during speech, acknowledging that lipreading accuracy can improve by focusing on the entire face, as speaking involves more than just the lips.also Table II provides a summary of key aspects from datasets mentioned in introduction. TABLE I : Wordset of persian dataset English Pronunciation English meaning Salam Hello Bro Go Bia Come Khodahafez Goodbye Surena Surena Begir Take Benevis Write Methodology TABLE II : Comparison of datasets Language Speakers Classes Total data resolution Urdu 10 18 2000 100\times 50 Arabic 73 10 1051 66\times 100 Indonesian 8 14 4000 640\times 480 Persian(ours) 20 7 2800 300\times 300 Persian 52 10 1560 26\times 44 This study explores two distinct approaches for lip-reading detection: an indirect method and a direct method. The indirect method relies on the tracking of facial landmarks to infer lip movements, while the direct method is anticipated to provide a more straightforward approach to data extraction and processing. The specifics of the direct method are still being finalized, but both approaches are crucial for understanding the overall efficacy of the lip-reading detection system. (a) (b) Fig. 2 : The structure of the (a) CNN and (b) LSTM Neural Networks in direct method I. Indirect Method In the indirect approach, lip-reading detection is based on the extraction and analysis of key facial landmarks over time. The dlib library is employed to identify and locate facial landmarks, specifically focusing on points 48 to 68, which correspond to the contour and shape of the lips. These landmarks are meticulously tracked across a series of video frames to capture the intricate movements of the lips during speech which can be seen in