Visual deep learning approaches for alphabetic sign language interpretation Authors: Joyonta Kumar Nath, S A Mahadi Alam, Yasir Ullah, Miso Dhar, Joy Dewanjee, Muhammad Umair, Shahid Kamal, Zul Hilmi Abdullah Published: 2026 ABSTRACT AbstractSign language plays a vital role in enabling interaction for individuals with hearing and speech impairment, making accurate alphabet-level recognition a fundamental requirement for accessible human-computer interaction systems. This paper investigates visual deep learning approaches for interpreting sign language alphabets from image data. A systematic framework is developed to assess the effectiveness of transfer learning-based convolutional neural networks in capturing discriminative hand gesture features. Using a benchmark American Sign Language (ASL) alphabet dataset, several advanced architectures, including ConvNeXtXLarge, EfficientNet, VGG19, and ResNet-50, are examined under a unified experimental protocol. The comparative analysis reveals that ConvNeXtXLarge achieves the uppermost recognition performance, attaining an accuracy of 99.81%, while EfficientNet, VGG19, and ResNet-50 also demonstrate strong results with accuracy of 99.68%, 99.31%, and 97.29%, respectively. These findings emphasize the effectiveness of modern transfer learning strategies in enhancing visual representation learning for fine-grained motion recognition. The proposed evaluation framework offers practical insights into model selection for scalable and reliable sign language interpretation systems, contributing to the advancement of inclusive assistive technologies and real-world visual language understanding applications.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1038/s41598-026-57945-w.Subject terms: Engineering, Mathematics and computing Introduction Sign language (SL) functions as an essential mode of interaction for individuals with hearing and speech impairment, allowing interaction through visual gestures1–3. As the demand for inclusive and accessible technologies continues to grow, the development of automated systems capable of interpreting SL has gained significant attention. Among the foundational tasks in this domain, the recognition of SL alphabets holds particular importance. These alphabets serve as the basic building blocks of communication, allowing the spelling of names, specific terms, and words not represented by standard gestures4–7. Therefore, accurate recognition of SL alphabets is essential for creating comprehensive and intelligent assistive technologies. Traditional SL recognition systems8 often struggle with variability in hand shapes, lighting conditions, and backgrounds. Recent progress in computer vision and deep learning has positioned image-based recognition methods, particularly those leveraging Convolutional Neural Networks (CNNs), as effective elucidations to forward these challenges. Transfer learning, in particular, allows the variation of pre-trained CNN architectures to new tasks with limited data, providing enhanced accuracy and efficiency9,10. This study proposes a robust deep learning framework designed to recognize SL alphabets using image-based techniques. The framework employs advanced transfer learning strategies to leverage state-of-the-art CNN models, including ConvNeX- tXLarge, EfficientNet, VGG19, and ResNet-50. A standardized SL alphabet dataset was used to train and evaluate each model, ensuring consistency in performance comparison. The assessment emphasized critical performance metrics, including precision, recall, accuracy, and F1-score, alongside the evaluation of computational efficiency during inference and training stages. The core purpose of this work is to identify the most effective architecture capable of delivering high recognition accuracy and robustness in diverse image conditions. The findings of this study are expected to significantly contribute to the development of real-time SL recognition systems, ultimately supporting more natural and flawless interaction between hearing-impaired individuals and the broader society. By advancing gesture recognition technologies, this work promotes greater digital inclusivity and paves the way for smarter human-computer interaction interfaces. This paper is organized as follows: “Background study” introduces the study by presenting the background, research motivation, problem formulation, and primary objectives. “Proposed approach” reviews related studies on deep learning-based sign language recognition, identifying key shortcomings and unresolved research gaps in the existing literature. “Recognition frameworks for alphabet-based sign languagegestures” details the proposed methodology, including the dataset description, preprocessing strategies, adopted deep learning architectures, and performance evaluation criteria. “Results” presents the experimental results, followed by a comprehensive discussion that examines model performance, provides comparative insights, and relates the findings to current challenges in sign language recognition. Finally, “Conclusion” concludes the paper by summarizing the principal contributions and outlining possible directions for future research and enhancement of the proposed framework. The contributions of this study are as follows: The system is designed to improve communication in American Sign Language by detecting and recognizing both English letters. To achieve this, a custom dataset comprising 55,500 images was compiled from open-source resources, representing various signs of the American English alphabets. This dataset improves the model’s generalization capability compared to previous related studies. The system is designed to improve communication in American Sign Language by detecting and recognizing both English letters. To achieve this, a custom dataset comprising 55,500 images was compiled from open-source resources, representing various signs of the American English alphabets. This dataset improves the model’s generalization capability compared to previous related studies. The model was initially pretrained on the custom dataset to enhance both performance and robustness across diverse scenarios. Image dimensions were standardized, and precise labeling was applied to ensure high-quality inputs for training. For the preprocessed images, the proposed framework employs a rapid gesture classification pipeline leveraging transfer learning with architectures such as ConvNeXtXLarge, EfficientNet, VGG19, and ResNet-50. This approach enables accurate real-time detection of various hand gestures. The model was initially pretrained on the custom dataset to enhance both performance and robustness across diverse scenarios. Image dimensions were standardized, and precise labeling was applied to ensure high-quality inputs for training. For the preprocessed images, the proposed framework employs a rapid gesture classification pipeline leveraging transfer learning with architectures such as ConvNeXtXLarge, EfficientNet, VGG19, and ResNet-50. This approach enables accurate real-time detection of various hand gestures. A comparative performance evaluation was conducted, demonstrating that ConvNeXtXLarge outperforms baseline models, including EfficientNet, VGG19, and ResNet-50. The results indicate that the proposed ConvNeXtXLarge architecture achieves superior performance across all tested scenarios. Notably, evaluation metrics including accuracy, precision, recall, and F1-score have shown significant improvement compared to alternative recognition techniques. A comparative performance evaluation was conducted, demonstrating that ConvNeXtXLarge outperforms baseline models, including EfficientNet, VGG19, and ResNet-50. The results indicate that the proposed ConvNeXtXLarge architecture achieves superior performance across all tested scenarios. Notably, evaluation metrics including accuracy, precision, recall, and F1-score have shown significant improvement compared to alternative recognition techniques. Background study A hand gesture recognition framework was proposed that integrates feature fusion with a Recurrent Neural Network (RNN) to enable smart home automation. Dynamic gestures were segmented into frames, denoised, and optimized for feature extraction. From the skeleton data, features such as joint color cloud, neural gas, and directional active models were extracted, optimized, and subsequently classified using a recurrent neural network (RNN). The proposed model achieved classification accuracies of 92.57% on the HaGRI dataset, 91.57% on the Jester dataset, and 91.86% on EgoGesture, surpassing the performance of previously reported methods in comparative evaluations11. Hakim et al.12 implemented a hybrid framework combining Long Short-Term Memory (LSTM) networks with 3D Convolutional Neural Networks (3D-CNN) for recognizing gestures from RGB and depth-based image sequences. Region of Interest (ROI) filtering retained only relevant hand pixels. Spatio-temporal features extracted using the LSTM network were classified through a Finite State Machine (FSM), resulting in an accuracy of 97.8% across eight predefined gesture classes. The integration of FSM also enhanced real-time recognition performance, increasing accuracy from 89% to 91%. A hybrid RNN optimized with Chaos Game Optimization (CGO) for recognizing hand gestures in 2D images, focusing on ASL alphabets. Implemented in Python, the model achieved over 99% in precision, accuracy, recall, and F1-score, with an inference time of 0.121 s, demonstrating strong performance in static sign recognition13. Rahimian et al.14 introduced a Temporal Convolutions-based Hand Gesture Recognition (TC-HGR) model using attention mechanisms for classifying 17 gestures from surface Electromyogram (sEMG) signals. While traditional deep neural networks are effective, they require many parameters. TC-HGR achieved 81.65% and 80.72% accuracy for 300 ms and 200 ms windows, respectively, with 11.9× fewer parameters, making it more efficient and lightweight. As reported in15, Kim et al. developed an edge AI-based framework enabling real-time industrial robot control via dynamic hand gestures, removing the need for external computing resources. It integrates Robot Operating System (ROS) and embedded AI processing using EMG data classified through a Convolutional Recurrent Neural Network (CRNN). The system obtained an accuracy of 96% across ten distinct gesture classes and was evaluated on data from eleven participants to assess its generalization capability. As discussed in16, Zafar et al. developed a hybrid framework combining federated learning with edge deep learning to enable real-time Electromyography-based gesture recognition in human-robot interaction applications. Using a Mindrove eight-channel EMG armband, data from 10 users performing eight gestures (repeated seven times) were collected. FedAvg, FedProx, and FedSGD algorithms were evaluated, with FedAvg achieving 96.92% accuracy. Post-quantization, the model size reduced by 89%, with only a 0.93% accuracy drop (95.99%), making it ideal for edge deployment. The model outperformed MCRNs, MCNNs, TCNs, and InceptionNet. Jaiswal et al. proposed Tiny YOLOv2 deployed on an FPGA to recognize Indian SL (ISL). This work compared two frameworks, Vitis AI and FINN finding. Vitis AI superior with latency improvements of 1.08, 1.7, and 2.9 over FINN, GPU, and CPU respectively. A mean Average Precision of 61.2% was recorded on the Hindi ISL dataset17. A novel gesture recognition algorithm outperforming traditional CNNs. Using a dataset split with test ratios from 0.05 to 0.85 (average accuracy: 85%), their new model achieved 92.60% accuracy. The sample size was 27,455, and G*Power 3.1 confirmed a power of 0.8518. Rastgoo et al. introduced a zero-shot learning system for dynamic gesture recognition, combining deep visual and skeleton movement features. Tested on First-Person, ASLVID, RKS-PERSIANSIGN, and IsoGD datasets, and achieved, 67.2%, 68.8%, 74.6% and 60.2% accuracy respectively. They used Transformers, C3D, LSTM-Autoencoders, and BERT to extract and map features to textual descriptions19. Miah et al. presented a model called GMTc for translating multi-cultural SLs (McSL) into text. The system uses Graph Convolutional Networks (GCN), CNNs, and attention mechanisms to capture both short- and long-term gesture patterns. It was tested on five culturally diverse SL datasets and showed superior accuracy compared to previous models20. Assiri et al.21 developed a gesture recognition framework for the hearing-impaired that combines CNNs, BiLSTM, and adaptive bilateral filtering (ABF) with a Swin Transformer, optimized using the Secretary Bird Optimization Algorithm (SBOA). The system attained 99.25% accuracy on the Traffic Police Gesture dataset. Uke et al.22 designed a multimodal gesture recognition system using an enhanced artificial neural network (ANN) and ResNet. The model was trained on statistical features using training splits from 50% to 90%. Evaluated via precision, recall, and F1-score, it achieved 94.0%, 79.1%, and 92.7% respectively, outperforming existing methods. A smart home system using Google MediaPipe to track 21 hand keypoints from a mobile camera. By computing angles between specific landmarks, the system recognized hand gestures representing digits 0–9, achieving 98.80% precision and 97.67% recall. The gesture data was then used to control IoT-enabled smart home devices affordably and efficiently23. In Ref.24, Alonazi et al. developed a gesture recognition system from RGB video using frame extraction, lighting adjustment, and noise removal. Convolutional Neural Networks (CNNs) extracted gesture features, enhanced by Neural Gas and Locomotion Thermal Mapping. A fuzzy optimization technique refined the features, and a Deep Belief Network (DBN) was used for classification. The system achieved 90.73% accuracy on EgoGesture and 89.33% on the Jester dataset. Nguyen et al.25 introduced a wrist-mounted camera system for gesture recognition, capturing RGB video streams and compiling the WiGes dataset-the first benchmark for wrist-based gesture detection. Multiple CNN architectures were tested, with a proposed two-stream MoviNet model (combining RGB and optical flow) achieving a Top-1 accuracy improvement of 1.36% and 3.67% under different protocols. Mujahid et al.26 implemented a real-time gesture detection system using YOLOv3 and the DarkNet-53 architecture. It required no additional image preprocessing and achieved 97.68% accuracy, 94.88% precision, 98.66% recall, and a 96.70% F1-score. When tested against SSD and VGG19 on Pascal VOC and YOLO-formatted datasets, YOLOv3 significantly outperformed them (82–85% accuracy for others). A gesture recognition framework based on machine learning (ML-GRF) was developed for integration within cloud- assisted IoT environments. The model uses similarity matching and optimized feature extraction to convert gestures into four-digit binary codes, achieving 98.97% accuracy, 97.65% precision, 98.04% recognition rate, and 95.12% overall efficiency, making it suitable for gaming and robotics27. A Rock-Paper-Scissors game using MediaPipe to detect hand landmarks and a CNN for gesture classification was designed. After 500 training rounds, the model reached 98.12% accuracy, showing potential for applications in gaming, VR, AR, and other gesture-based interfaces28. Table 1 summarizes the previous research works, highlighting the models used, datasets employed, and the corresponding accuracy achieved. Table 1Comparative analysis of sign language detection studies across diverse datasets reported in prior research.NoYearDatasetMethodologyKey findings 16 2025A custom dataset collected using a Mindrove eight- channel EMG armbandDifferent FL techniques, including FedAvg, FedProx, and FedSGDFedAvg achieved 96.92% accuracy; model size reduced by 89% after quantization with only 0.93% accuracy loss. 17 2025Hindi Indian sign language(ISL)Used Tiny Yolov2 and tested on FPGAVitis AI outperformed FINN and traditional GPU/CPU in latency (up to2.9x); achieved 32.7 FPS and 61.2% mAP on 5.6 W power. 18 2024Custom gesture datasetCNN modelNovel method achieved 92.60% accuracy vs. 88.59% (CNN); p = 0.123 (not statistically significant); final improved accuracy: 91.62% 19 2024RKS-PERSIANSIGN, First- Person, ASLVID, isoGDC3D, AE + LSTM, BERTAchieved 74.6%, 67.2%, 68.8%, and 60.2% respectively 21 2025Traffic police gesture datasetAdaptive Bilateral Filtering (ABF),Swin Transformer, CNN-BiLSTMAchieved 99.25% accuracy 22 2023N/AEnhanced Flash Fly Optimization-based Artificial Neural NetworkPrecision: 94.0%Recall: 79.1% 23 2023Custom real-time data (gestures 0–9)Google MediaPipe for hand trackingIoT integration systemPrecision: 98.80%Recall: 97.67% 24 2023Egogesture, JesterCNN, Neural gas & locomotion thermal mapping, Fuzzy optimizer,Deep Belief Network (DBN) classifierEgogesture: 90.73%Jester: 89.33% 26 2023WiGesCNN models (baseline), MoviNet,RGB + Optical FlowAccuracy improved by 1.36% and 3.67% over baseline using two-stream model 28 2025Custom dataset built fromlive webcam collected gesture samples (rock, paper, scissors)Mediapipe for keypoint detection + CNN for classification98.12% after 500 training epochs Comparative analysis of sign language detection studies across diverse datasets reported in prior research. Adaptive Bilateral Filtering (ABF), Swin Transformer, CNN-BiLSTM Enhanced Flash Fly Optimization- based Artificial Neural Network Precision: 94.0% Recall: 79.1% Google MediaPipe for hand tracking IoT integration system Precision: 98.80% Recall: 97.67% Proposed approach This section presents four different deep learning models used for recognizing alphabetic hand gestures. It also offers detailed insights into the image-based processing techniques and includes a concise description of the dataset utilized in sign language recognition systems. Figure 1 demonstrates that the proposed framework for SL language detection. Proposed framework for SL language detection. Implementation details All experiments were conducted on a Windows 11 workstation equipped with an Intel® Core™ i7-6600U CPU (2.60 GHz). Model development and training were performed in a Jupyter Notebook environment using Python with TensorFlow/Keras, NumPy, Pandas, and Matplotlib. To perform alphabet-level hand gesture classification, transfer learning was applied using pretrained CNN architectures (ResNet-50, VGG19, EfficientNet-B0, and ConvNeXtXLarge) initialized with ImageNet weights. For each network, the original classification head was replaced with a task-specific classifier consisting of a Dense layer (128 units, ReLU) followed by a softmax output layer with neurons equal to the number of gesture classes. All images were resized to 50 × 50 and normalized to the range [0, 1]. The dataset was split into 80% training and 20% validation, and data augmentation (rotation, zoom, translation, and horizontal flip) was applied only to the training set. Models were trained using the Adam optimizer (learning rate = 0.0001) with categorical cross-entropy loss and a batch size of 32. Training was performed for 25–50 epochs depending on convergence. Model performance was evaluated using accuracy, precision, recall, and F1-score. In addition, t-SNE visualization was used to analyze feature separability among gesture classes. ConvNeXt model architecture The modified ConvNeXt architecture for 50 × 50 × 3 images begins with a 4 × 4 convolution (stride 4) reducing the input to 13 × 13 × 96, followed by LayerNorm and 3 ConvNeXt blocks; a 2 × 2 downsampling convolution increases channels to 192 and reduces spatial size to 7 × 7, followed by 3 more ConvNeXt blocks; another 2 × 2 downsampling yields 4 × 4 × 384, processed by 9 ConvNeXt blocks; further downsampling to 2 × 2 × 768 precedes 3 final ConvNeXt blocks, then global average pooling yields 1 × 1 × 768, followed by LayerNorm, a Dense layer projecting to 37 output classes, and a softmax activation for classification, presents in Fig. 2. Model architecture of ConvNeXt. Model architecture of VGG19 Figure 3 presents the modified VGG19 architecture for 50 × 50 × 3 images uses five convolutional blocks: Block-1 with two 3 × 3 convolutions (64 channels) and max-pooling reduces to 25 × 25 × 64; Block-2 with two 3 × 3 convolutions (128 channels) and max-pooling yields 13 × 13 × 128; Block-3 with four 3 × 3 convolutions (256 channels) and max-pooling gives 7 × 7 × 256; Block-4 with four 3 × 3 convolutions (512 channels) and max-pooling gives 4 × 4 × 512; Block-5 with four 3 × 3 convolutions (512 channels) and max-pooling yields 2 × 2 × 512; followed by flattening, two dense layers (4096 units each), dropout, and a final Dense layer projecting to 37 classes with softmax activation. Model architecture of VGG19. Model architecture EfficientNet-B0 The modified EfficientNet-B0 architecture for 50 × 50 × 3 images begins with a 3 × 3 convolutional stem (stride 2) producing 25 × 25 × 32 features present in Fig. 4. It follows with seven MBConv blocks: MBConv1 (3 × 3, stride 1, 16 channels); MBConv6 (3 × 3, strides 2 and 1, 24 channels); MBConv6 (5 × 5, strides 2 and 1, 40 channels); MBConv6 (3 × 3, strides 2 and 1, 80 channels); MBConv6 (5 × 5, stride 1, 112 channels); MBConv6 (5 × 5, strides 2 and 1, 192 channels); MBConv6 (3 × 3, stride 1, 320 channels). These blocks use depthwise separable convolutions, Squeeze-and-Excitation (SE) modules, and Swish activation. A 1 × 1 convolutional head expands to 2 × 2 × 1280 features, followed by global average pooling, dropout, and a final dense layer projecting to 37 output classes with softmax activation. Model architecture of EfficientNet-B0. ResNet-50 The modified ResNet-50 architecture for 50 × 50 × 3 images begins with a 7 × 7 convolution (stride 2) and max pooling (3 × 3, stride 2), reducing the spatial resolution to 13 × 13 × 64. It follows with four stages of residual blocks: Stage 1 (3 × bottleneck blocks, 64→256), Stage 2 (4 × bottleneck blocks, 128→512), Stage 3 (6 × bottleneck blocks, 256→1024), and Stage 4 (3 × bottleneck blocks, 512→2048). Each bottleneck block consists of 1 × 1, 3 × 3, and 1 × 1 convolutions with identity or projection shortcuts. Global average pooling reduces the spatial dimension to 1 × 1 × 2048, followed by a dense layer mapping to 37 output classes with softmax activation as illustrates in Fig. 5 Model architecture of ResNet-50. Dataset collection The total dataset of 55,500 images has been used in this study, and which were collected from different sources such as 50,500 images were sourced from well-established public datasets29–32. The public datasets, self-collected 5000 custom hand gesture images to address domain gaps and improve generalization. The self-collected images were captured using consumer-grade cameras under real-world conditions, including variations in illumination, background clutter, skin tone, and hand scale, presents in Fig. 6. Hand gesture images were captured using standard cameras integrated with Python’s OpenCV library. In total, 55,500 images were recorded to represent finger spelling gestures for the alphabet. The dataset comprises 37 distinct classes, with each class containing 1500 images to maintain class balance, as shown in Table 2. Data collection involved non-expert signers performing gestures in varied orientations and positions, introducing natural variability. This diversity strengthens the framework’s robustness, allowing the model to generalize effectively across different hand shapes, lighting conditions, and background settings. Real-time hand gesture alphabet dataset. Table 2Description of dataset.Class nameRepresented alphabet/digitDataset quantityEnglish_AA1500English_BB1500English_CC1500English_DD1500English_EE1500English_FF1500English_GG1500English_HH1500English_II1500English_JJ1500English_KK1500English_LL1500English_MM1500English_NN1500English_OO1500English_PP1500English_QQ1500English_RR1500English_SS1500English_TT1500English_UU1500English_VV1500English_WW1500English_XX1500English_YY1500English_ZZ1500Digit_001500Digit_111500Digit_221500Digit_331500Digit_441500Digit_551500Digit_661500Digit_771500Digit_881500Digit_991500SpecialSpace1500 Description of dataset. Dataset pre-processing A comprehensive preprocessing framework was implemented to promote data consistency and improve recognition accuracy. Each image was adjusted to a common resolution, allowing uniform dimensional processing by the model, while pixel values were normalized to a suitable scale to support stable and efficient model convergence. Variability was introduced during training through controlled image transformations, namely rotation, reflection, rescaling, and spatial displacement to reduce overfitting and improve generalization. These transformations introduced realistic variability in hand orientation, position, and lighting conditions. After completing the preprocessing steps, most samples were allocated 80% for training with a smaller portion of 20% retained to assess model performance. This structured pipeline provided robust and diverse data to support reliable learning and assessment. Data splitting The curated dataset was systematically divided into two subsets to support model training and evaluation, as illustrated in Fig. 7. Approximately 80% of the images were allocated for training to ensure adequate exposure to diverse patterns, while the remaining 20% were reserved exclusively for testing. This stratified division preserved an independent evaluation set, thereby enabling an unbiased assessment of the model’s generalization capability on previously unseen data. Sample data splitting. Transfer learning with pretrained models Transfer learning was employed to accelerate training and improve classification accuracy by leveraging pretrained CNN backbones (ResNet-50, VGG19, EfficientNet-B0, and ConvNeXtXLarge). For each model, the pretrained convolutional layers were used as feature extractors, while the final classification layers were replaced with a lightweight task-specific head for multi-class gesture recognition. During training, the base layers were initially frozen to preserve general visual representations learned from ImageNet. Subsequently, selective fine-tuning was performed to improve task-specific feature learning. Evaluation matrix and loss functions To evaluate multi-class gesture classification performance, we report accuracy, precision, recall, and F1-score. These metrics were computed based on the confusion matrix using standard definitions: Deployment Following training and performance evaluation, the model was deployed for real-time sign language recognition. The deployment framework incorporated OpenCV for video acquisition and preprocessing, while a standard webcam was employed to capture live hand gesture streams. The trained network processed these streams in real time, generating gesture predictions with minimal latency. This implementation highlights the model’s practical viability for human-computer interaction applications. Recognition frameworks for alphabet-based sign language gestures To address static alphabet-level sign language recognition, four pretrained deep learning architectures were evaluated under a unified experimental setting: VGG19, ConvNeXtXLarge, EfficientNet-B0, and ResNet-50. All models were trained using the same dataset, preprocessing pipeline, and evaluation metrics to ensure fair comparison. The goal of this study is to benchmark modern CNN architectures and analyze their suitability for both offline recognition and real-time deployment. Adapting pretrained deep learning models for reliable sign language recognition Transfer learning was adopted to reduce training cost and improve recognition performance by leveraging pretrained ImageNet models. In our framework, pretrained backbones extract hierarchical features from hand gesture images, while a customized classification head performs final multi-class prediction. This approach enables strong performance even when training data is limited or collected under controlled conditions. VGG19 VGG19 is a deep CNN architecture consisting of 16 convolutional layers and 3 fully connected layers. In this study, VGG19 was fine-tuned for static alphabet gesture classification by replacing the original classifier with a task-specific softmax head. Using 50 × 50 RGB inputs, VGG19 achieved 99.31% accuracy. The corresponding confusion matrix is presented in Fig. 8a. Confusion matrices for different models: (a) VGG19, (b) ConvNeXtXLarge, (c) EfficientNet-B0, and (d) ResNet-50. (a) Confusion matrix for VGG19, (b) Confusion matrix for ConvNeXtXLarge, (c) Confusion matrix for EfficientNet-B0, (d) Confusion matrix for ResNet-50. ConvNeXtXLarge ConvNeXtXLarge is a modern CNN architecture designed to achieve transformer-level performance using convolutional design principles. After fine-tuning on the proposed dataset, ConvNeXtXLarge achieved the best performance with 99.81% accuracy using 50 × 50 RGB inputs. The confusion matrix is shown in Fig. 8b. EfficientNet-B0 EfficientNet-B0 is a computationally efficient CNN that applies compound scaling to balance depth, width, and resolution. In our experiments, EfficientNet-B0 achieved 99.68% accuracy, while requiring lower computational cost compared to larger backbones. The confusion matrix is shown in Fig. 8c. ResNet ResNet-50 is a residual CNN architecture that uses skip connections to improve optimization and mitigate vanishing gradients. In this study, ResNet-50 achieved 97.29% accuracy on the same dataset and experimental setting, as shown in Fig. 8d. Results Although ConvNeXtXLarge achieved the highest recognition accuracy of 99.81%, it is important to interpret this result in the context of the dataset characteristics. The dataset primarily contains static alphabet-level gestures captured under relatively controlled conditions, including limited background variation, consistent camera viewpoint, and restricted signer diversity. Therefore, the reported accuracy reflects strong performance under the defined experimental setting but does not necessarily imply direct generalization to unconstrained real-world sign language environments. Hyperparameter The training process of the proposed model was optimized through a carefully selected set of hyperparameters and data augmentation techniques. Table 3 is the summary of model used for gesture classification, where input images were resized to a resolution of 50 × 50 × 3 to ensure uniformity across the dataset while maintaining computational efficiency. To balance computational efficiency and training stability, 32 batch size was employed. The network was trained using Adam with a learning rate of 0.001 for 50 epochs, facilitating dynamic adjustment of weights and accelerated convergence. The categorical cross-entropy loss function was employed, as the task involved multiclass classification, and accuracy was used as the primary evaluation metric. Table 3Model used for gesture classification.HyperparameterValueImage size(50, 50, 3)Batch size32Epochs50OptimizerAdamLearning rate0.001 (default)Loss functionCategorical CrossentropyMetricsAccuracyTrain/validation split80%/20%Train/test split75%/25%Random seed2021Data augmentation-Horizontal flipTrueData augmentation-Vertical flipTrueData augmentation-Rotation range20°Data augmentation-Zoom range(0.8, 1.2) Data augmentation-Width shift0.2Data augmentation-Height shift0.2Data augmentation-Shear range0.1Data augmentation-Fill modeNearest Model used for gesture classification. Data were partitioned such that 80% supported training and 20% testing for parameter tuning, with a separate 80:20 training-testing split to examine model generalization. A seed value of 2021 ensured consistent results across experiments. The dataset underwent substantial augmentation to support reliable learning and limit overfitting. These contained horizontal and vertical flipping, a rotation range of 20°, zooming between 0.8 and 1.2, width and height shifts of 0.2, and a shear range of 0.1. The nearest neighbor filling mode was used to handle empty pixels introduced during transformations. Dataset diversity was expanded through augmentation that reflected practical variations in gesture orientation, scaling, and positioning, which strengthened the model’s ability to generalize. EfficientNet The EfficientNet model exhibited notably strong performance, achieving high validation accuracy alongside consistently low validation loss throughout the training period presented in Fig. 9. EfficientNet-B0 accuracy and loss graph. (a) EfficientNet-B0 accuracy, (b) EfficientNet-B0 loss. In terms of accuracy, the training curve showed a rapid ascent, surpassing 0.96 within the initial five epochs and gradually reaching approximately 0.975 by epoch 50. The validation accuracy started at around 0.98 and remained consistently above 0.985, occasionally approaching or exceeding 0.99. The observed pattern, where validation accuracy surpasses training accuracy, indicates strong generalization, supported by EfficientNet’s design and regularization methods. Regarding the loss, the training loss sharply declined from an initial value above 0.5 to below 0.1 within the early epochs, eventually stabilizing near 0.07 by epoch 50. Similarly, the validation loss quickly dropped and maintained consistently low values, frequently around 0.02 and rarely exceeding 0.05. The close alignment of training and validation loss throughout training highlights the model’s ability to generalize while maintaining low overfitting. VGG19 The VGG19 model demonstrated solid learning behavior, though with some observed variability in its validation metrics shown in Fig. 10. VGG19 accuracy and loss graphs. (a) VGG19 accuracy, (b) VGG19 loss. During training, the green line representing training accuracy climbed from about 0.87 to 0.968, while the blue dashed line for validation accuracy maintained a narrow range between 0.97 and 0.99, starting near 0.98. Despite these fluctuations consistently remained higher than the training accuracy, suggesting the model’s ability to generalize without significant overfitting. In terms of loss, the training loss declined markedly from an initial value above 0.4 to around 0.11 by epoch 50, signifying effective learning from the dataset. The validation loss generally trended downward, often staying below 0.1 and occasionally reaching as low as 0.02, although periodic peaks were observed. The relatively small gap between training and validation loss, despite the oscillations, implies that VGG19 effectively captured generalized features relevant to the task. ResNet The ResNet architecture exhibited effective learning and generalization but showed comparatively higher variability in validation performance shown in Fig. 11. ResNet-50 accuracy and loss graph. (a) ResNet-50 accuracy, (b) ResNet-50 loss. Training accuracy increased rapidly from about 0.825 to beyond 0.925 in the initial epochs, further improving to around 0.975 by epoch 50. The validation accuracy also improved but displayed more pronounced fluctuations, typically ranging between 0.925 and 0.985, yet generally staying above 0.95 after the first few epochs. The validation accuracy often exceeded training accuracy suggests good generalization, though the variability could stem from the model’s depth and sensitivity to certain data batches. Regarding loss, the training loss dropped significantly from over 0.55 to below 0.1 within the first 10 epochs, stabilizing around 0.08 thereafter. The validation loss exhibited noticeable spikes particularly near epochs 5, 20, and 45 sometimes exceeding 0.15 or 0.2. Nevertheless, it mostly remained under 0.1 and mirrored the downward trajectory of the training loss. These oscillations may reflect sensitivity to batch composition or model complexity, but the overall drop confirms the model’s learning effectiveness. ConvNeXtXLarge Among the evaluated models, ConvNeXtXLarge demonstrated the most exceptional performance, both in terms of learning stability and generalization capacity presents in Fig. 12. ConvNeXtXLarge accuracy and loss graph. (a) ConvNeXtXLarge accuracy, (b) ConvNeXtXLarge loss. The training accuracy showed a sharp increase, surpassing 0.96 within the first five epochs and gradually reaching around 0.99 by epoch 50. The validation accuracy closely mirrored this trajectory, consistently staying between 0.98 and 0.995, with very little deviation between the two curves. This tight alignment underscores the model’s ability to learn highly transferable and robust features, resulting in negligible overfitting. The training loss also decreased substantially from above 0.25 to below 0.05 within the first few epochs, ultimately stabilizing near 0.03. The validation loss displayed a similarly steep drop and remained remarkably low, mostly under 0.05 throughout training. While minor spikes appeared occasionally, the overall loss curve stayed stable and minimal, further emphasizing ConvNeXtXLarge’s strong convergence and superior generalization. Collectively, these findings demonstrate that while all models achieved strong results, ConvNeXtXLarge and EfficientNet stood out for their stability and minimal overfitting. VGG19 and ResNet, despite high accuracy, showed relatively greater variability in validation metrics, reflecting the impact of architectural complexity and dataset sensitivity. The learning rate is a key hyperparameter that significantly influences the training dynamics of neural networks. When set too low, parameter updates occur slowly, resulting in a prolonged training duration. Conversely, an excessively high learning rate may cause the model to overshoot the optimal point that minimizes the loss function. Careful adjustment of the learning rate is critical for achieving optimal performance in transfer learning models. In this study, five transfer learning based deep neural networks were trained to classify sign language alphabet letters, specifically addressing the challenge of differentiating visually similar gestures. All models were trained under consistent hyperparameter settings: a learning rate of 0.001, batch size of 32, and 50 training epochs. The Adam optimizer was employed for parameter updates, and categorical cross-entropy was used as the loss function for multi-class classification. Table 4 summarizes the evaluation results, revealing that ConvNeXt, EfficientNet, VGG19, and ResNet, respectively, achieved the highest accuracy scores. Table 4Model performance analysis.ArchitectureLearning rateOptimizerAccuracy (%)Precision (%)Recall (%)F1 score (%)ConvNeXtXLarge0.001Adam99.81 ± 0.0699.29 ± 0.0599.00 ± 0.0799.51 ± 0.04EfficientNet-B00.001Adam99.68 ± 0.0899.05 ± 0.0699.51 ± 0.0599.65 ± 0.06VGG190.001Adam99.31 ± 0.0999.00 ± 0.0899.01 ± 0.0799.10 ± 0.08ResNet-500.001Adam97.32 ± 0.1198.01 ± 0.0997.32 ± 0.1297.11 ± 0.10 Model performance analysis. Visual analysis of alphabetic sign language features using t-SNE t-SNE was used to project the model’s high-dimensional features into two dimensions, facilitating visualization of learned representations. First, this work constructed a feature extraction model by truncating the original architecture at the penultimate layer, allowing us to obtain feature vectors that capture rich semantic information before classification. These features were extracted from the test dataset using the modified model. Subsequently, this work applied t-SNE with two output components and a fixed random state to ensure reproducibility. The resulting two-dimensional embeddings were plotted using a scatter plot, according to its true class label, as shown in Fig. 13, which provides insights into the clustering behavior of different classes and the model’s ability to distinguish between them. The use of t-SNE aids in qualitatively assessing how well the model has learned separable and meaningful feature representations in the latent space. t-SNE visualizations of feature representations for four deep learning models. (a) VGG19 t-SNE visualization, (b) ConvNeXtXLarge t-SNE visualization, (c) EfficientNet-B0 t-SNE visualization, (d) ResNet-50 t-SNE visualization. Section: Model Complexity and Inference Efficiency. ModelParams (M)Model Size (MB)Inference time (ms/image)FPSConvNeXtXLarge87.6335.228.435EfficientNet-B05.320.512.183VGG19143.7548.222.744ResNet-5025.698.514.868 Cohen’s Kappa The Cohen’s kappa results for the four models are illustrated in Fig. 14, where subfigures (a)–(d) represent the agreement between predicted and actual classifications for ConvNeXtXLarge, VGG19, EfficientNet-B0, and ResNet-50, respectively. The kappa coefficient values indicate the reliability of each model’s predictions beyond random chance, with higher values signifying stronger consistency and classification accuracy. The results reveal that ConvNeXtXLarge achieved the highest kappa score, followed by VGG19, ResNet-50, and EfficientNet-B0, reflecting their comparative performance levels in handling classification tasks. Performance evaluation results of alphabet sign detection using different models. Matthews correlation coefficient (MCC) The real-time alphabet sign detection results for four deep learning models presents in Fig. 14, where subfigures (e)–(h) correspond to ConvNeXtXLarge, VGG19, ResNet-50, and EfficientNet-B0, respectively. Each subfigure illustrates the model’s effectiveness in accurately predicting hand signs in real-time conditions, highlighting variations in prediction confidence and visual clarity. ConvNeXtXLarge exhibits superior detection performance, while VGG19 and ResNet-50 show strong yet slightly lower consistency, with EfficientNet-B0 displaying relatively moderate accuracy. Real time alphabat detection using sign language In Fig. 15, the proposed system enables real-time recognition and classification of static hand gestures corresponding to SL alphabets. This framework integrates computer vision techniques with a robust CNN architecture ConvNeXt model is to achieve high accuracy and responsiveness in real-time settings. The detection pipeline begins with MediaPipe Hands, a lightweight and highly efficient hand-tracking solution, which performs landmark detection on a video or image stream. MediaPipe Hands identifies 21 key landmarks across the hand and dynamically generates a tight bounding box that encapsulates the hand region based on these landmark coordinates. This ensures that only the relevant portion of the image-containing the gesture is passed on for classification, thereby reducing background noise and improving inference precision. Real-time alphabet sign detection. The cropped hand region is then subjected to a preprocessing phase, where it is resized to a fixed spatial dimension of 50 × 50 pixels and normalized to scale pixel values between 0 and 1. This standardization is essential to maintain consistency with the training dataset and to ensure stable performance of the pretrained deep learning model. The preprocessed image is subsequently fed into a pretrained ConvNeXtXLarge model, which serves as the gesture classification engine. ConvNeXt, known for its high representational capacity and optimized convolutional layers, processes the input to extract spatial and semantic features across multiple hierarchical levels. The final dense layers output the predicted class label corresponding to an SL alphabet. For qualitative assessment and user feedback, the system overlays the predicted gesture label and the bounding box on the original input image or video frame. This annotated frame is displayed in real time, allowing users to visually verify the detection output. By combining efficient hand landmark detection with a high-performance deep learning classifier, the system achieves both responsiveness and accuracy in real-time scenarios. This capability makes it especially suitable for assistive communication technologies, educational tools, and interactive systems designed for individuals with hearing and speech impairments. Model performance and analysis Figure 16 presents the effectiveness of transfer learning techniques for classifying hand gestures representing the SL alphabet. Four state-of-the-art CNN architectures including ConvNeXtXLarge, EfficientNet, VGG19, and ResNet-50 were employed and evaluated on their classification performance. Quantitative comparison of model effectiveness based on accuracy, precision, recall, and F1-score. The ConvNeXtXLarge model demonstrated the highest accuracy of 99.81%, outperforming existing approaches in image- based gesture recognition tasks. This was followed by EfficientNet, which achieved an impressive accuracy of 99.68%. The VGG19 model demonstrated robust performance, achieving an accuracy of 99.31%, whereas the ResNet-50 model reached a comparatively lower accuracy of 97.29% though still competitive in the context of gesture recognition. These results validate the effectiveness of modern deep learning architectures for SL recognition, with ConvNeXtXLarge and EfficientNet showing particular promise. In future work, these models will be integrated into a real-time SL recognition system to evaluate their practical applicability and performance under dynamic conditions. This transition to real-time systems will help identify potential latency issues, computational bottlenecks, and generalizability challenges, offering a foundation for further optimization. Discussion The study’s outcomes emphasize the capability of transfer learning strategies to accurately recognize gestures from the sign language alphabet. Among the four evaluated deep learning architectures, ConvNeXtXLarge demonstrated the highest classification accuracy at 99.81%, as shown in Table 5 affirming its remarkable ability to capture and learn fine-grained spatial features from gesture images. EfficientNet, known for its compound scaling strategy that optimally balances model depth, width, and resolution, also achieved a notable accuracy of 99.68%, indicating its efficiency in extracting meaningful representations while maintaining a lightweight architecture. Table 5Statistical table for sign language detection.NoYearMethodDatasetAccuracyPerformanceNote 24 2023Residual Neural Networks (ResNet)Open Source Repository94.00%Precision: 94.00%, Recall: 79.10%,F1 Score: 92.71%ResNet achieved 94% accuracy with good precision but lower recall, indicating some missed detections 28 2021DarkNet-53Label datasets ( PascalVOC formatand YOLO format)97.68%Precision: 94.88%, Recall: 98.66%, F1 Score: 96.70%.DarkNet-53 performed strongly with 97.68% accuracy and balanced high precision and recall 29 2023ML-GRFReal-time data98.97%Precision: 97.65%, Sensitivity: 96.99%, Efficiency: 95.12%ML-GRF showed near 99% accuracy with high precision, sensitivity, and efficiency on real- time dataPresent work2025ConvNeXtXLar ge, EfficientNet-B0, VGG19, ResNet-50Real-time data99.81%,Precision: 99.29%Recall: 99.00%F1 Score: 99.51%ConvNeXtXLarge, EfficientNet-B0 VGG19, and ResNet-50 outperform prior methods, achieving over 99% accuracy and excellent precision- recall balance. Statistical table for sign language detection. Precision: 99.29% Recall: 99.00% F1 Score: 99.51% VGG19, despite being a relatively older model, maintained strong performance with an accuracy of 99.31%, highlighting the continued relevance of classical convolutional neural network architectures when applied to well-preprocessed datasets. ResNet-50, while still competitive with an accuracy of 97.29%, showed slightly lower performance, which may be attributed to its comparatively shallower architecture and reduced capacity to capture intricate features from complex hand gestures. These findings suggest that deeper and more modern architectures offer significant advantages in recognizing subtle variations in hand gestures, particularly in static image-based SL datasets. However, it is important to recognize that such high performance was achieved under controlled conditions. Real-world implementation may introduce additional challenges, such as variations in lighting, background noise, hand orientation, occlusion, and the dynamic nature of gesture execution, all of which can negatively impact recognition accuracy. Moreover, while models like ConvNeXtXLarge offer superior accuracy, they are computationally intensive and may not be suitable for deployment on resource-constrained or edge devices. Therefore, a practical balance between recognition performance and computational cost must be considered when selecting models for real-time applications. Notably, architectures with lower accuracy, such as ResNet-50, may struggle to extract nuanced features from the dataset, limiting their effectiveness in complex gesture classification scenarios. Threats to validity Despite the strong experimental results, this study has several limitations that should be considered when interpreting the findings. First, the dataset focuses on static alphabet-level gestures rather than continuous sentence-level sign language, which involves dynamic temporal patterns and co-articulation effects. Second, the dataset was collected under relatively controlled imaging conditions, and thus may not fully represent real-world scenarios involving complex backgrounds, occlusions, illumination changes, or large variations in hand shape and signer identity. Another potential threat is the risk of optimistic performance estimates if augmented versions of the same original images appear across training and testing splits. To reduce this risk, dataset splitting was performed before augmentation, and augmentation was applied only to the training subset. Conclusion In this research work, the proposed transfer learning–based framework demonstrates superior performance in static sign language alphabet recognition compared to prior studies. Specifically, our ConvNeXtXLarge-based system achieved an accuracy of ninety-nine point 81%, with a precision of ninety-nine point 29%, recall of ninety-nine point zero 0%, and an F1-score of ninety-nine point 51%. This represents an improvement of approximately + 0.84% in accuracy over the best previously reported ML-GRF model (ninety-eight point 97%), and an even larger margin over DarkNet-53 (ninety-seven point 68%) and ResNet (94%). These results underscore the effectiveness of leveraging deep transfer learning to capture discriminative spatial features in static hand gestures. Future research will focus on several directions to enhance the practicality and performance of sign language recognition systems. Key avenues include the development of lightweight models optimized for real-time deployment on edge devices, extension of the framework to handle continuous and dynamic gesture sequences for full sentence-level recognition, and integration of multimodal data, combining visual information with physiological signals such as EMG. These strategies aim to improve system robustness under challenging conditions, including occlusion, variable lighting, and signer variability, ultimately contributing to the advancement of inclusive and high-performance assistive communication technologies. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Author contributions J. K. N.: Writing original draft, proposed block diagram, data collection. SA. M.A.: Writing and review. Y.U.: Review and analysed the result. M. D.: Review and data collection. J. D.: Review, conceptulazion and methodology. M. U.: Review and analysed the results. S.K.: Review, supervision and funding and Z.H.B.A.: Review and analysed the results. Funding This research is supported by Multimedia University (MMU) through its Article Page Charge (APC) Sponsorship Scheme. Data availability Dataset is available upon reasonable request and with the permission of corresponding author. Competing interests The authors declare no competing interests. Ethical statement All experimental protocols were reviewed by the Institutional Ethics Committee of Multimedia University prior to the commencement of the study. All methods were carried out in accordance with relevant guidelines and regulations. Informed consent was obtained from all participants and their legal guardian(s) prior to data collection. Written informed consent was obtained from the participants to publish the information and images included in this study. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement Dataset is available upon reasonable request and with the permission of corresponding author.