LSC50: Colombian Sign Language Video and Inertial Measurement dataset Authors: Andrés Felipe Flórez-Sierra, Brayan David Solórzano, Fredy Segura-Quijano, Yenny Milena Cortés-Bello, Luis Cubillos, Luis Felipe Giraldo, Christian Cifuentes-De la Portilla Published: 2024 ABSTRACT In Colombia, approximately more than 500.000 people have disabling hearing loss, representing around 1% of the population in Colombia, and only 400 professional interpreters of Colombian Sign Language (LSC) are available in the whole country. In this data descriptor, we present a dataset with recordings of Colombian Sign Language (LSC) as an important effort to develop technologies that facilitate social inclusion and equity for the deaf community in Colombia. This dataset includes multimodal recordings of native and non-native signers signing different elements of the LSC lexicon. In particular, we provide recordings of video and inertial measurement units, along with detected body landmarks and computed information on the inverse kinematics of the upper body such as flexion, extension, and rotation of arms and wrists. This dataset is the first of its kind for LSC and it is key for the development of technologies needed to better understand the language and reduce communication barriers between people with and without hearing loss. Background & Summary Currently, more than 500.000 have disabling hearing loss in Colombia, where around 70% of this population lives in poverty and 15% never attended elementary school. Most hospitals do not have interpreters of the Colombian Sign Language (LSC), and the emergency dispatch office can only be contacted by phone. Lack of communication is a critical issue that marginalizes this community. Even though research on LSC began around 20 years ago when researchers from universities and deaf associations started to study the linguistic functions of LSC and its teaching and learning aspects1, there is still a lot to do to bridge the gap between the deaf and hearing communities. In pursuing advancing technologies that enhance communication between individuals with and without hearing impairments, numerous databases housing corpora for various sign languages have emerged. Among these, American Sign Language (ASL) stands out as one of the most extensively studied2. These datasets encompass a range of media including videos, images, and recordings captured by Kinect cameras, offering a comprehensive insight into the sign language2–4. Beyond ASL, datasets for other sign languages like Turkish5 (226 signs, includes videos recorded only by Kinect V2 camera), Polish6 (84 gestures recorded by Kinect V2 cameras), and Arabian Sign Language7 (32 signs and about 54.000 images) have also been compiled. Some others, such as the dataset for hand gesture recognition (HGR)8 offer a valuable resource for Polish and American Sign Language gesture recognition research. This dataset includes RGB images, binary skin presence masks, and hand landmark annotations. In Latin America, two significant datasets have emerged: LSA64 for Argentinian Sign Language9, containing images and video recordings of 64 signs, and a corpus for Mexican Sign Language, focused on the creation of an automatic translator10. All those related datasets available in the literature are based on videos or images. No other information is collected related to the motion required to generate a sign, independently of the language. There is an important gap exists in the integration of alternative motion data such as accelerometers into these datasets. Particularly for the Colombian Sign Language (LSC), there is a lack of research aimed at developing tools for automated analysis beneficial to linguistics and sign language recognition. Moreover, open datasets containing annotated recordings of authentic LSC interpreters, essential for advancing this field, are currently lacking. While some studies have made efforts to identifying letters of the LSC alphabet from static hand images of non-sign language speakers11–13, incorporating the dynamic elements of sign language comprising structure, grammar, vocabulary, and gestures with temporal information from both hands and faces, is imperative for comprehensive understanding and technology development. This data descriptor presents the first open dataset14 that includes multimodal recordings of LSC signers that can be useful for the analysis and development of technologies for the analysis and recognition of LSC. As a consequence of the multimodal nature of LSC, the dataset we present has recordings from different sources of information representing a set of 50 signs of LSC. We provide a total of 4000 RGB and RGB-D videos of the signers, along with more than 500 face and body landmarks. Also, the dataset has inertial information on the movement of the signs captured through inertial measurement units (IMU), which facilitates movement analysis and its reconstruction through inverse kinematics and is adjusted to be used with OpenSim. With the LSC50 dataset14 and its variety of data types, we intend to develop new tools for the analysis and recognition of LSC. For example, it can be used to develop AI-based sign language recognition systems, utilizing multimodal data such as videos, inertial measurements, and body landmarks for real-time translation of Colombian Sign Language (LSC). The dataset is valuable for gesture analysis and motion tracking, supporting technologies in virtual reality and human-computer interaction (HCI) systems aimed at accessible interfaces. Linguistic researchers can study the structural and semantic aspects of LSC, while educational platforms can create interactive tools for teaching and assessing sign language skills. Additionally, its biomechanical data, including inverse kinematics, supports human motion studies, contributing to a deeper analysis of movement patterns in sign language. Methods Participants were recruited through the Association of Colombian-Spanish Sign Language Interpreters. Two of them are deaf and three of them are not hearing impaired. All participants are fluent in Colombian Sign Language and were above 18 years old at the time of their participation in the data collection process. All data, videos, and images collected in the project were dissociated from the participants’ personal information. Standard labels were generated to encode the files, thus guaranteeing the confidentiality and privacy of the data. Fifty representative signs of LSC were selected from which groupings can be generated by semantic, contextual, and meaning relationships to facilitate linguistic, thematic, concept, or attribute analysis. The study of the different representations of LSC from the groups mentioned above allows the understanding of the origin and multimodal nature of LSC. Consequently, the selected LSC signals can be grouped and related as shown in Fig. 1. It shows a possibility of grouping the signs chosen and listed in Table 1. These signs are categorized by semantic relationships such as hypernyms, hyponyms, polysemy, and synonymy. The selected signs correspond to semantic fields of tentatively universal concepts for lexicostatistical purposes in sign languages. These were taken from a basic dictionary of Colombian Sign Language, which accounts for an average basic lexicon, with an emphasis on useful and commonly used vocabulary in the urban context. Therefore, “PEOPLE” can be considered a common sign and semantically related to all the others. From there, seven possible semantic groupings are recognized and demarcated by colors in the graph. These groupings range from family members to descriptive signs of feelings or sensations, representing a semantic relationship of hypernyms. There are also semantic groupings by hyponyms, from the signs associated with personal treatment to the grouping of basic questions.Fig. 1Graph that represents the list of 50 signs chosen for the dataset as described in Table 1, in which word groups are made by means of semantic relationships and meaning or concept.Table 1List of signs included in the LSC50 dataset.Sign-SpanishSign-EnglishSign-SpanishSign-EnglishSign-SpanishSign-EnglishGRACIASTHANK YOUTÍOUNCLEBIENVENIDOWELCOMEBUENOS DÍASGOOD MORNINGHERMANOBROTHERPERDÓNSORRYBUENAS TARDESGOOD EVENINGHAMBREHUNGERPERMISOPERMISSIONBUENAS NOCHESGOOD NIGHTFELIZHAPPYNUNCANEVERSEÑASIGNCONTENTOGLADYOMENOMBRENAMETRISTESADTÚYOUTRABAJARTO WORKABURRIDOBOREDUSTEDESYOUCOMERTO EATBIENGOOD¿QUÉ?WHAT?VIVIRTO LIVEMALWRONG¿CUÁNDO?WHEN?POCOLITTLE BIT¿CÓMO ESTÁS?HOW ARE YOU?¿DÓNDE?WHERE?FAMILIAFAMILYMÁS O MENOSMORE OR LESS¿CÓMO?HOW?PERSONASPEOPLESENTIRFEEL¿POR QUÉ?WHY?MUJERWOMANJUCIOSOJUCIOS¿QUIÉN?WHO?HOMBREMENHOLAHELLODIFERENTEDIFFERENTNIÑOBOYADIÓSBYETODOSALLNIÑAGIRLPOR FAVORPLEASEMUCHOMUCHABUELOGRANDFATHERCON GUSTOWITH PLEASUREThis table presents the immediate translation into Spanish and English. Graph that represents the list of 50 signs chosen for the dataset as described in Table 1, in which word groups are made by means of semantic relationships and meaning or concept. Table 1List of signs included in the LSC50 dataset.Sign-SpanishSign-EnglishSign-SpanishSign-EnglishSign-SpanishSign-EnglishGRACIASTHANK YOUTÍOUNCLEBIENVENIDOWELCOMEBUENOS DÍASGOOD MORNINGHERMANOBROTHERPERDÓNSORRYBUENAS TARDESGOOD EVENINGHAMBREHUNGERPERMISOPERMISSIONBUENAS NOCHESGOOD NIGHTFELIZHAPPYNUNCANEVERSEÑASIGNCONTENTOGLADYOMENOMBRENAMETRISTESADTÚYOUTRABAJARTO WORKABURRIDOBOREDUSTEDESYOUCOMERTO EATBIENGOOD¿QUÉ?WHAT?VIVIRTO LIVEMALWRONG¿CUÁNDO?WHEN?POCOLITTLE BIT¿CÓMO ESTÁS?HOW ARE YOU?¿DÓNDE?WHERE?FAMILIAFAMILYMÁS O MENOSMORE OR LESS¿CÓMO?HOW?PERSONASPEOPLESENTIRFEEL¿POR QUÉ?WHY?MUJERWOMANJUCIOSOJUCIOS¿QUIÉN?WHO?HOMBREMENHOLAHELLODIFERENTEDIFFERENTNIÑOBOYADIÓSBYETODOSALLNIÑAGIRLPOR FAVORPLEASEMUCHOMUCHABUELOGRANDFATHERCON GUSTOWITH PLEASUREThis table presents the immediate translation into Spanish and English. List of signs included in the LSC50 dataset. This table presents the immediate translation into Spanish and English. Given the multimodal nature of sign language, which involves dynamic and static gestures of the body in combination with facial expressions, it is necessary to understand the movement patterns from a visual, dynamic, and spatial perspective to give meaning to the representation of the sign language. Obtaining a time-based data record of the nature of the gestures involved in sign language is essential for an accurate understanding of these movement patterns. A complete description of the movement and expression of the sign language is chosen to obtain visual, inertial, and descriptive information. Therefore, in the visual section, motion capture is performed through RGB cameras from different perspectives, focusing on facial expression and whole-body movement. Additionally, depth video information is added to understand better the three-dimensional movement associated with signs. On the other hand, the spatial and dynamic information associated with movement can be studied from an inertial perspective of crucial points of the human body involved in signs. From this information, it is possible to reconstruct the body’s movement with inverse kinematics methods. Finally, from the visual information obtained, descriptive and simplified representations of body movement and facial expression can be obtained through landmarks in crucial points of the body and face. Therefore, estimating the pose of fundamental body parts in the meaning of the signs and their respective analysis can be carried out through time series of the landmarks. Ethics statement With the approval of our respective Institutional Review Board at Universidad de los Andes, supporting the ethical integrity of our research(IRB number 723), we recruited participants who signed an informed consent. This consent described the nature of the study, extended a voluntary and free invitation to participate, and granted permission to publish the collected data in open repositories, including face and body images and videos. Names or personal data at any project stage were not revealed. Video data acquisition In the visual section, information on movement patterns is acquired by recording four simultaneous videos of the same sign from different perspectives and objectives. The first video corresponds to an RGB video of the face of the LSC speaker taken with a Nikon D5600 DSLR camera with an AF-P DX NIKKOR lens with a focal length of 55 millimeters. A diaphragm opening of 3.5 G, video that records at a rate of 50 frames per second (FPS) at a resolution of 640 by 424 pixels Fig. 2a. The second type of video corresponds to an RGB capture of the LSC speaker’s body from the knees to a height of maximum vertical extension of the arms. This type of video is recorded at a rate of 24 FPS, and a resolution of 1920 by 1080 pixels Fig. 2b. The third type corresponds to a grayscale video representing the scene’s depth. This video is recorded at a rate of 30 FPS at a resolution of 512 by 424 pixels Fig. 2c. Finally, the next type of video records a grayscale video associated with the intensity of infrared radiation (IR) detected by the sensor. This video is recorded at the same rate and resolution as the one representing the depth Fig. 2d. For RGB full body, depth, and IR videos, the Microsoft Kinect V2 is used, which has RGB sensors, infrared emission, and reception that work at the mentioned rates and resolutions.Fig. 2(a) RGB video capture focused on the face. (b) Full-body focused RGB video capture. (c) Grayscale video capture of the depth of the scene. (d) Grayscale video capture representing IR of the scene.(e) OpenSim representation of the scene generated from the IMU data. The participant provided consent to show her face in this figure. (a) RGB video capture focused on the face. (b) Full-body focused RGB video capture. (c) Grayscale video capture of the depth of the scene. (d) Grayscale video capture representing IR of the scene.(e) OpenSim representation of the scene generated from the IMU data. The participant provided consent to show her face in this figure. Regarding the spatial distribution of the recording set, a group of cameras is located at a distance of 200 centimeters from the LSC speaker, which is 10 centimeters from a blue curtain of dimensions three by three meters. The height of the Nikon d 5600 DSLR camera is adjusted to eye level for the LSC speaker, and the Microsoft Kinect V2 sits 5 centimeters below the DSLR camera, where both are docked on a 3D printed platform and mounted on a tripod that allows adjusting the height and inclination in the three axes. The lighting must be white and homogeneous. Inertial measurement data acquisition The inertial measurement data acquisition was carried out using the XSENS XDOT sensors located in seven key points of the body, as seen in Fig. 3. These points are the center of mass of the humerus, the radius, the carpus, and the lower thoracic region (vertebra T12). These locations were chosen due to existing protocols of body motion analysis models involving these points15. In addition, the IMU XSENSE XDOT sensors allow the measurement of gyroscope, accelerometer, and magnetometer, which includes: Orientation (Expressed in quaternions), Change of orientation (dq change over time of the quaternions and dv speed in m/s of the sensors in the three main axes), Acceleration (Linear acceleration in the three main axes in m/s2), Angular speed (angular speed on the three main axes in deg/s) and Magnetic field (Measurement of the magnetic field in the three principal axes expressed in units normalized to the Earth’s magnetic field). All data has a sampling rate of 120 hertz at a resolution of 16 bits for each type of unit. The XSENS XDOT sensors present the orientation output in Unit normalized Quaternions which by default are in East-North-Up earth frame: q0, (i)q1, (2)q2, (k)q3 or w, x, y, z. In general, for this IMU, the local earth-fixed reference coordinate system is defined as a right-handed Cartesian coordinate system with x positive to the East, y positive to the North and z positive when pointing up. The accuracy of the sensors is 1.0° for dynamic activities and the gyroscope has a typical scale factor variation of 0.5%.Fig. 3Location of the IMU XSENSE XDOT sensors on the human body. Hands, forearms, upper arms, and upper trunk. The sensors are located at the center of mass of their respective body segments. Location of the IMU XSENSE XDOT sensors on the human body. Hands, forearms, upper arms, and upper trunk. The sensors are located at the center of mass of their respective body segments. Inverse kinematics Using the data provided by the XSENSE XDOT sensors, the inverse kinematics was calculated using a generic model of the human body available in OpenSim/OpenSense. This simulation tool allows us to estimate the motion angles based on the IMU data in quaternions (Please, see Fig. 4d)16,17. From the inverse kinematics, it is possible to obtain a motion representation that enables studying the physical actions related to sign language, which involves the results of flexion, extension, and rotation of the arms and wrists.Fig. 4(a) Hand pose estimation performed in full body focused RGB video. (b) Face mesh Estimation performed in face-focused RGB video. (c) Pose estimation performed in full body focused RGB video. (d) Inverse kinematics validation in OpenSim calculated using the IMU data. The participant provided consent to show his face in this figure. (a) Hand pose estimation performed in full body focused RGB video. (b) Face mesh Estimation performed in face-focused RGB video. (c) Pose estimation performed in full body focused RGB video. (d) Inverse kinematics validation in OpenSim calculated using the IMU data. The participant provided consent to show his face in this figure. Video landmarks acquisition From the RGB videos presented in the video data acquisition section, descriptive information about the signs made by the LSC speaker can be obtained from the estimation of critical points known as landmarks located in different regions of the body. For this, the holistic models provided by the MediaPipe framework are used. MediaPipe Holistic uses the pose, face, and hand cue point models in MediaPipe Pose, MediaPipe Face Mesh, and MediaPipe Hands, respectively, to generate a total of 543 cue points; 33 pose landmarks, 468 face landmarks, and 21 reference landmarks in each hand. Pose estimation A list of reference points of the body pose. Each reference point consists of x, y, and y reference coordinates normalized to [0.0, 1.0] times the width and height of the image, respectively. Face mesh Estimation A list of 468 facial landmarks. Each reference point consists of x, y, and z coordinates. The reference points x and y are normalized to [0.0, 1.0] by the image’s width and height, respectively. z represents the reference point depth, with the depth at the center of the head being the origin, and the smaller the value, the closer the reference point is to the camera. The magnitude of z is roughly the same scale as x and y. Hand pose estimation A list of 21 benchmarks for each hand. Each reference point consists of x, y, and z. The reference points x and y are normalized to [0.0, 1.0] by the image’s width and height, respectively. z represents the reference point depth with the depth at the wrist as the origin, and the smaller the value, the closer the reference point is to the camera. The magnitude of z is roughly the same scale as x and y. Signals data synchronization Synchronization of the recorded data from both the IMUs and the cameras was achieved by introducing pauses between the signals and starting and finishing each capture in the same position. Following each signal, the subject lowers their hands to a fixed position and remains there for a short period. The timestamps used to delineate each IMU signal are recorded in the database, located in the “IMU/INFO/” folder. The cameras were initialized at the same time using a Python script. This strategy guaranteed same-time video registration. This procedure ensures that the data is effectively synchronized with the videos, which are already organized and categorized by signal in the database. Data Records We provide public access to the dataset and all code related to this work. The LSC50: Colombian Sign Language Video dataset14 explores the specific case of the recognition of 50 signs defined in the Colombian sign language listed in Table 2. The dataset14 has 4000 videos (.avi format) distributed in RGB, depth, and IR. Additionally, it has inertial measurement information involved in the movement of each of the signs. Moreover, finally, landmark files (.csv format) associated with the RGB videos were obtained. The total information in the dataset14 is equally distributed in the number of classes (50 signs) and the number of participants involved (5 volunteers: 3 native speakers and two non-native speakers).Table 2Set of 50 signs of the Colombian sign language considered in the dataset.NSignNSignNSignNSignNSign0000GRACIAS0010FAMILIA0020FELIZ0030HOLA0040USTEDES0001BUENOS DÍAS0011PERSONAS0021CONTENTO0031ADIÓS0041¿QUÉ?0002BUENAS TARDES0012MUJER0022TRISTE0032POR FAVOR0042¿CUÁNDO?0003BUENAS NOCHES0013HOMBRE0023ABURRIDO0033CON GUSTO0043¿DÓNDE?0004SEÑA0014NIÑO0024BIEN0034BIENVENIDO0044¿CÓMO?0005NOMBRE0015NIÑA0025MAL0035PERDÓN0045¿POR QUÉ?0006TRABAJAR0016ABUELO0026¿CÓMO ESTÁS?0036PERMISO0046¿QUIÉN?0007COMER0017TÍO0027MÁS O MENOS0037NUNCA0047DIFERENTE0008VIVIR0018HERMANO0028SENTIR0038YO0048TODOS0009POCO0019HAMBRE0029JUCIOSO0039TÚ0049MUCHON corresponds to the numerical label established in the dataset and Sign corresponds to the most immediate translation into Spanish. Table 2Set of 50 signs of the Colombian sign language considered in the dataset.NSignNSignNSignNSignNSign0000GRACIAS0010FAMILIA0020FELIZ0030HOLA0040USTEDES0001BUENOS DÍAS0011PERSONAS0021CONTENTO0031ADIÓS0041¿QUÉ?0002BUENAS TARDES0012MUJER0022TRISTE0032POR FAVOR0042¿CUÁNDO?0003BUENAS NOCHES0013HOMBRE0023ABURRIDO0033CON GUSTO0043¿DÓNDE?0004SEÑA0014NIÑO0024BIEN0034BIENVENIDO0044¿CÓMO?0005NOMBRE0015NIÑA0025MAL0035PERDÓN0045¿POR QUÉ?0006TRABAJAR0016ABUELO0026¿CÓMO ESTÁS?0036PERMISO0046¿QUIÉN?0007COMER0017TÍO0027MÁS O MENOS0037NUNCA0047DIFERENTE0008VIVIR0018HERMANO0028SENTIR0038YO0048TODOS0009POCO0019HAMBRE0029JUCIOSO0039TÚ0049MUCHON corresponds to the numerical label established in the dataset and Sign corresponds to the most immediate translation into Spanish. Set of 50 signs of the Colombian sign language considered in the dataset. N corresponds to the numerical label established in the dataset and Sign corresponds to the most immediate translation into Spanish. The dataset14 contains three folders characterized by the type of information they contain: video, inertial measurement, and landmarks. The dataset14 structure is represented in Fig. 5.Fig. 5Dataset structure. The figure shows the distribution of the subfolders and their respective number of files. Dataset structure. The figure shows the distribution of the subfolders and their respective number of files. Videos The first folder contains the 4000 videos in.avi format evenly distributed in four subfolders. The first subfolder contains RGB videos focused on the participant’s face. The second contains RGB video of the participant’s body (from the knees to the maximum vertical extension). The third contains grayscale videos of the depth of the scene. Moreover, the fourth contains a grayscale video representing the scene’s infrared. Each video has a tag XX _XXXX _XXXX _XXXX where each X corresponds to a digit. The first four digits symbolize the class number (Sign N presented in the Table 2), the second four digits correspond to the number of the volunteer, and the last four digits represent the number of the repetition of the sign made. The same labels are found in the four video subfolders and represent a simultaneous shot of the four types of videos. An example of tag 0001_0002_0003 is interpreted as sign number 1, volunteer 2, and repetition number 3. IMU The IMU folder contains information on 250 inertial measurements associated with the 50 signs considered. The information is distributed in four subfolders, INFO, STO, RAW, and OUT_OPENSIM. The INFO subfolder has information related to the participants and the location of the IMU sensors, as well as the timestamps related to each signal. Inside, there are three types of files: a file with a.osim extension (OpenSim models), and several text files. The OpenSim file contains the musculoskeletal model used to calculate the inverse kinematics from the inertial sensors and each text file (.txt) has the anthropometric measurements of the upper limbs of each subject. In the STO subfolder, there are files with the extension “.sto” that contain the orientation data generated by the IMU sensors asociated to each body segment. These data were structured in a way that can be used in OpenSim software. The file header specifies the frequency of the data, the data type (quaternion), and the version of OpenSim. Following the header, there is a table that contains the combination of quaternions for each body segment during the entire motion capture Inside the RAW subfolder, there are several files in the.xlsx format that hold the raw output data of the IMU sensors. Each table contains multiple sheets named after the body segments where the IMU sensor was placed. These sheets contain the values for Quaternions, derivatives, Gyroscope, accelerometer, and magnetometer for each plane. The fourth subfolder OUT_OPENSIM, contains the information of the inverse kinematics process carried out by OpenSim The .mot file can be loaded directly into an OpenSim model to visualize the movement. If contains kinematic information in angles. It is required that the OpenSim model includes the upper limbs to be able to represent the sign language motion. The information stored in the STO, RAW, and OUT_OPENSIM subfolders has the same tag structure as the one mentioned for the video folder. Finally, in order to use the data from inertial sensors in OpenSim, the data available in the RAW folder must be prepared. This data contains information related to the motion such as orientation and acceleration that are required to import the data to OpenSense, the OpenSim package for using inertial sensors. This package requires the.sto file included in the STO folder. It contains the orientation information obtained from the sensors, associated with each body segment. With these files, OpenSense places the sensors in each segment of the biomechanical model and allows the calculation of the inverse kinematics by generating the.mot file, which can be found in the OUT_OPENSIM folder. These files contain kinematic data related to the motion required for the signs recorded. For more details, please see the “OpenSense - Kinematics with IMU Data” guide18. Landmarks The Landmarks folder contains information on the landmarks associated with the videos in the dataset14. This folder contains three subfolders, each one with files in .csv format that store coordinates of points of interest on each video frame. The first subfolder contains the landmarks of the participant’s face. Therefore, the videos associated with these landmarks are RGB videos focused on the face. The second subfolder contains the landmarks associated with the participant’s body pose. Moreover, the third subfolder contains the landmarks associated with the pose of the participant’s hands. For the second and third subfolders, the videos associated with the landmarks correspond to the RGB videos of the participant’s body. All folders contain files separated by commas, which have the position in X, Y, and Z for each landmark. Each file follows the same tag format as the video folder. The .csv file headers follow the format: landmark_i_x, landmark_i_y, and landmark_i_z, where ‘i’ represents the index number of each extracted landmark’s coordinates in three-dimensional space. Each landmark number is assigned according to the MediaPipe results for Body, Hand, and Face. Accessory data files To complement the presented dataset14, a.osim file associated with a generic human body model is added to be processed in the OpenSim platform. This model has the pre-defined location of the inertial measurement units so that a load of data housed in the /IMU/STO folder is loaded to the program on time. Also, a Python file is added that allows processing of the depth videos’ RAW information to reduce the image’s noise and increase the detail associated with the difference in depth of the objects in the recorded scene. Technical Validation Sign validation To guarantee the fidelity of the representation of the signs made and stored in the dataset14, certified members of the National Federation of Colombian sign language interpreters provided participation and supervision at each data acquisition stage. These LSC interpreters are non-native speakers and are some of the participants that can be seen in the stored data. In addition, these interpreters guaranteed communication with the native speakers who participated in the project. Kinematics validation Body movement kinematics associated with the inertial measurements captured by the XSENSE XDOT sensors were validated by comparing results obtained from the OpenSim model with the motion registered with the optical tracking motion system available in our lab. The validation process involved comparing basic arm movements (flexion, abduction, and elbow flexion) recorded using both inertial sensors and an optical tracking system. For validation, several repetitions of each movement type were performed, and the averages were calculated. OpenSim recordings with the inertial measurement units (IMUs) were conducted at a sampling rate of 120 Hz, while the optical system operated at 240 Hz. To compare the curves, the Pearson correlation coefficient was calculated using the average curves for each movement, as can be seen in Fig. 6. The correlation in all cases was high; the minor variations can be attributed to OpenSim employing an idealized generic model with a neutral static reference position.Fig. 6Validation process performed to this dataset. An optical motion tracking system was used to compare kinematics results obtained from OpenSim using the IMU data. The curves show the average of the motion registered from each type of movement. The Pearson correlation coefficient was calculated for each case. Validation process performed to this dataset. An optical motion tracking system was used to compare kinematics results obtained from OpenSim using the IMU data. The curves show the average of the motion registered from each type of movement. The Pearson correlation coefficient was calculated for each case. The captures were conducted simultaneously to facilitate synchronization. To address the differing sampling rates of the devices, Lagrangian polynomial interpolation was employed to compare data from both systems. Usage Notes Researchers can use LSC50 to assess video classification associated with Colombian Sign Language and motion inertial detection using new architectures to detect sign language patterns automatically. Additionally, the dataset can be used to explore new methods to understand sign language based on biomechanic characteristics such as range of motion, kinematics, and energy, among others. Limitations The representation of the movement obtained from the inertial measurements of the XSENSE DOT sensors and subsequently processed in the Simtk-OPENSIM software are subject to the body model that is considered in the use of the software. Therefore, if the model used for inverse kinematics does not match the body dimensions of the participant from whom the measurements were obtained, a less-than-perfect representation of the movement will be obtained. Therefore, the fidelity of the movement representation is highly dependent on the configuration of the body model used in the software5,19–30.