Emotion recognition in dance therapy driven by DanceEmoNet: a deep learning model based on facial expression and pose estimation Authors: Yadan Ye Published: 2026 ABSTRACT With the widespread application of dance therapy in mental health interventions, precise, real-time identification of emotional fluctuations has become a significant challenge. Existing emotion recognition methods predominantly rely on unimodal information, making it difficult to fully capture the complexity and temporal dependencies of emotional changes. To address this, the DanceEmoNet model is proposed, integrating facial expression recognition and pose estimation techniques to improve the precision of emotion recognition through multimodal feature fusion. The model employs YOLOv11 for face and pose detection, utilizes a hybrid TriBAN network and CNN-LSTM model for feature extraction and temporal modeling, and performs feature fusion via the GCNC module. Experimental comparisons with several benchmark models demonstrate that DanceEmoNet achieves better results across multiple metrics, exhibiting faster inference speed (e.g., reduced per-frame latency) and lower computational cost (e.g., fewer FLOPs), with overall performance gains ranging from 5 to 10%. The experimental results confirm the clear strengths of DanceEmoNet in capturing complex emotional changes and dynamic dance movements, indicating its practical applicability for real-world deployment. Introduction In today's society, mental health issues have increasingly become a global focus, especially when dealing with emotions such as stress, anxiety, and depression. Traditional treatment methods often fail to meet the growing and diverse needs of individuals. Dance therapy, as a body-mind integrated approach, has gained widespread attention in recent years (Zhang and Wei, 2024). However, emotional recognition in dance therapy remains a challenging task. Accurately capturing and analyzing the emotional changes of individuals during the therapeutic process has become one of the key difficulties in this field of research (Dieterich-Hartwell et al., 2025). With the development of deep learning technologies, emotion recognition has made significant progress in many fields, particularly in facial expression and pose estimation, offering new solutions (Sun and Wu, 2023). Nevertheless, existing methods still face challenges in terms of recognition accuracy and real-time performance, especially in complex dynamic dance therapy scenarios. Therefore, proposing new solutions for emotion recognition in dance therapy holds significant theoretical and practical value (Turab et al., 2025; Nasa and Das, 2026). Currently, emotion recognition technology is widely applied in various fields, including psychology, medicine, education, and security. Facial expression-based emotion recognition methods, such as DeepFace and VGG-Face, have made significant progress and are extensively used in emotion analysis (Onim et al., 2025; Ning et al., 2026). These methods capture facial muscle changes to accurately identify different emotional states. Physiological signal analysis methods, such as ECG-ER and fNIRS-ED, provide an alternative approach to emotion recognition by analyzing physiological signals. VER and MEA have become research hotspots, particularly for predicting and assessing emotional states in adolescents (Wang et al., 2024; Shou et al., 2023). PBR and BER offer new perspectives for emotion recognition (Zhang et al., 2023; Martinez-Martin and Fernández-Caballero, 2025). However, the application of these methods in dance therapy faces challenges related to real-time performance and complexity in dynamic environments, and they have yet to accurately capture emotional changes in complex dance movements (Garg and Kaur, 2026). Under the framework of deep learning, emotion recognition methods based on facial expressions and pose estimation have gradually become a key research focus (Si et al., 2025). The application of CNNs and RNNs in image and video data has greatly advanced emotion recognition technology (Agung et al., 2024; Manalu and Rifai, 2024). The introduction of 3D Pose Estimation techniques has made pose analysis an important direction for emotion recognition (Sun and Wu, 2023). Although these methods have shown good results in static images, they still struggle to handle complex dynamic scenarios in dance therapy, which involve significant movement changes and intricate emotional expressions. Methods combining facial expressions with speech for emotion recognition have gradually been proposed; however, in the context of dance therapy, the processing of unimodal data still faces significant limitations (Wu et al., 2025). In the field of emotion recognition, many studies focus on different data modalities, exploring ways to improve the accuracy of emotion recognition (Cao et al., 2025). In facial expression recognition, DCNN and FERDL are widely used to perform emotion classification by extracting deep features from facial expression images (Elsheikh et al., 2024; Zhao et al., 2015). In pose estimation, PoseFlow and OpenPose are common methods for pose analysis, which analyze emotional changes by capturing keypoint positions and motion trajectories (Xiu et al., 2018; Cao et al., 2019). Gait recognition is employed to capture emotional states through gait analysis (Li et al., 2025). SSL is used to learn emotion representations from unlabeled data, generating high-quality emotion feature representations through self-supervised learning methods (Ning et al., 2025). Although these methods have achieved some success in their respective fields, their application in the complex dynamic context of dance therapy still faces challenges, especially when dealing with complex dance movements and real-time emotion recognition tasks (Sun and Wei, 2025). To overcome this limitation, this study proposes a deep learning model based on facial expression and pose estimation—DanceEmoNet. By combining these two modalities, the model enhances the accuracy and real-time performance of emotion recognition, adapting to the complex and dynamic emotional expressions in dance therapy. The study presents the following three contributions: A multimodal feature fusion method based on facial expression and pose estimation is proposed, addressing the challenge of emotion recognition in complex dance movements.A highly efficient hybrid deep learning model is designed, which maintains high recognition accuracy while meeting real-time performance requirements.The effectiveness of the DanceEmoNet model is validated through experiments, demonstrating its practical application potential in dance therapy. A multimodal feature fusion method based on facial expression and pose estimation is proposed, addressing the challenge of emotion recognition in complex dance movements. A multimodal feature fusion method based on facial expression and pose estimation is proposed, addressing the challenge of emotion recognition in complex dance movements. A highly efficient hybrid deep learning model is designed, which maintains high recognition accuracy while meeting real-time performance requirements. A highly efficient hybrid deep learning model is designed, which maintains high recognition accuracy while meeting real-time performance requirements. The effectiveness of the DanceEmoNet model is validated through experiments, demonstrating its practical application potential in dance therapy. The effectiveness of the DanceEmoNet model is validated through experiments, demonstrating its practical application potential in dance therapy. The structure of this paper is as follows: Section 2 introduces the dataset, experimental details, and evaluation metrics; Section 3 explains the overall design of the model and the implementation of its sub-models; Section 4 presents and analyzes the experimental results; Section 5 summarizes the contributions and limitations of the study and outlines directions for future research. Theory method Datasets In emotion recognition and pose estimation research, dataset selection is critical, particularly for scenarios involving complex emotional and motion changes. This paper primarily uses two publicly available datasets: FER-2013 and Human3.6M. However, these datasets may not fully reflect affective movement in therapeutic settings. FER-2013 contains low-resolution, posed faces, and Human3.6M focuses on laboratory motion capture data without emotion labels, which restricts their direct applicability to therapeutic environments. To address this limitation, additional experiments were conducted using the RECOLA dataset, which includes multimodal data such as facial expressions, speech, body posture, and physiological signals. This dataset better aligns with emotion recognition and dynamic movement analysis in therapeutic contexts. Table 1 presents the basic characteristics of FER-2013 and Human3.6M, along with the addition of RECOLA in the experiments. Table 1Dataset overview: basic information of FER-2013, Human3.6M, and RECOLA datasets.DatasetTypeSamplesEmotion classesApplicationImage sizeFER-2013Facial expression recognition35,887Seven emotionsEmotion classification48 × 48Human3.6MPose estimation4,000No emotion classPose estimation320 × 240Action recognitionRECOLAMultimodal emotion recognition1,000Continuous emotion scoresEmotion and affective dynamics analysisVaries Dataset overview: basic information of FER-2013, Human3.6M, and RECOLA datasets. The FER-2013 dataset primarily consists of facial expression images from individuals around the world, with each image depicting various emotional responses. Each image in the dataset is labeled with the emotion it portrays, such as anger, disgust, sadness, surprise, happiness, etc., (Tutuianu et al., 2024). Due to the dataset's relatively low image resolution and basic emotion categories, it provides a practical basis for training and validating emotion recognition models. However, the FER-2013 dataset, like many emotion recognition datasets, focuses primarily on expressive style and does not capture the full range of complex emotional states, especially those that are blended or context-dependent. During the data preprocessing stage, the images are first standardized to meet the neural network input requirements. Next, facial alignment algorithms are applied to adjust the position and pose of the facial region, ensuring accurate expression extraction. Data augmentation—including rotation, scaling, and lighting changes—is then performed to improve the model's stability under varying conditions. The Human3.6M dataset focuses on pose estimation and action recognition tasks, containing multiple 3D motion capture data, which covers various poses and actions from different individuals (Zhu et al., 2023). During the data preprocessing phase, denoising is necessary since the raw data may contain some noise. Afterward, the data is standardized to ensure input stability. For the pose estimation task, the labeled keypoint information is used to calculate the spatial coordinates of the actions. These data are often utilized to generate 3D skeletal models for further pose analysis. Based on this, the model can extract features from each action and combine them with facial expression information in emotion recognition, forming a complete emotional state prediction system. However, similar to the FER-2013 dataset, the Human3.6M dataset does not provide emotional labels, and emotional states inferred from pose data alone may be limited in capturing the full spectrum of affective expressions, particularly in dynamic dance therapy settings. The RECOLA dataset focuses on multimodal emotion recognition and dynamic affective state analysis, providing continuous emotion scores along with facial expressions, speech, body posture, and physiological signals, such as heart rate and respiration rate (Shoer and Erzin, 2025). It is specifically designed to analyze emotions in contexts where both facial and physiological responses are important, making it ideal for therapeutic settings such as dance movement therapy. During the data preprocessing stage, multimodal data synchronization is crucial to align facial, speech, and physiological signals. The facial expression data undergoes facial alignment to standardize the pose, while the speech data is processed through feature extraction techniques to obtain pitch and intensity characteristics. The physiological signals are normalized to ensure consistency across the dataset. To improve the model's performance and robustness, data augmentation techniques such as time-shifting and adding noise are applied to the multimodal data. This dataset is valuable for training emotion recognition models in dynamic and complex therapeutic environments, where emotional responses often vary and interact with physical movements. However, it should be noted that while the RECOLA dataset provides more contextually rich emotion data, expert-annotated affect and participant self-reports would further enhance the model's ability to detect genuine emotional states in therapeutic dance contexts. This will be explored in future work to better capture the full range of affective states. Experimental details In the experiments conducted in this paper, all tests were performed on a high-performance computer to ensure efficient processing of large-scale datasets, particularly for the training and inference of deep learning models. The hardware configuration used in the experiments includes: an NVIDIA A100 GPU (40GB VRAM), an Intel Xeon Gold 6248R CPU (20 cores), 128GB DDR4 memory, and 2TB SSD storage. The powerful computational capacity of the GPU significantly accelerates the training process of deep learning models, especially when handling the computationally intensive DanceEmoNet model, effectively reducing training time and improving experimental efficiency. The Intel Xeon Gold processor supports parallel multi-task computing, providing strong support for the computation-heavy tasks during model training, ensuring the smooth progression of the training process. The operating system used is Ubuntu 20.04 LTS, and the deep learning frameworks are PyTorch 1.8 (Facebook's AI Research lab (FAIR), Menlo Park, California, USA) and TensorFlow 2.0 (Google Brain team in Mountain View, California, USA), combined with CUDA 11.0 and cuDNN 8.0, to ensure efficient utilization of the GPU for accelerated computation. The Python version is 3.8 (Python Software Foundation, Beaverton, Oregon, USA), ensuring compatibility with all deep learning frameworks and their dependencies. The facial images in the FER-2013 dataset were standardized and enhanced using data augmentation techniques to enlarge data variation. All images were resized to 48 × 48 pixels and converted to grayscale to maintain consistent input dimensions. FER-2013 includes 35,887 anonymized samples collected from a large number of individuals across diverse backgrounds, providing wide sampling of facial expressive patterns without subject-specific identifiers. In this dataset, the ground truth labels correspond to discrete emotion categories provided by the original dataset annotations and were directly used as supervision signals for model training and evaluation. For FER-2013, 80% of the data was used for training and 20% for testing. To investigate the model's performance under increasing emotional complexity, the data were further organized according to different emotion transition patterns, such as neutral-to-happy, neutral-to-sad, and transitions between negative emotions, while retaining the original emotion labels as ground truth for each frame. For the Human3.6M dataset, the anonymized 3D pose data underwent noise removal, calibration, and interpolation to ensure temporal continuity, followed by normalization of pose coordinates to a unified scale. Human3.6M contains motion capture recordings from 11 anonymized subjects performing various actions under controlled laboratory conditions. As Human3.6M does not provide explicit emotion annotations, ground truth emotion labels were derived by aligning pose sequences with externally defined emotion transition scenarios, which were used consistently across all models for comparative evaluation. Action sequences were temporally segmented, with 70% of the data used for training and 30% for testing, and balanced sampling applied across action categories. In addition, the RECOLA dataset was incorporated to provide a more therapeutically relevant evaluation setting. RECOLA consists of anonymized multimodal recordings from 27 subjects, including synchronized facial expressions, speech, body movements, and physiological signals, along with continuous emotion annotations. In this dataset, the ground truth is defined by continuous affective ratings provided by human annotators, representing the temporal evolution of emotional states. These continuous labels were used as reference signals to evaluate the model's predictions under gradually evolving emotional dynamics. During preprocessing, multimodal signals were temporally aligned and normalized, and a subject-independent split was adopted, with approximately 70% of subjects used for training and 30% for testing to assess cross-subject generalization. Although this study relies exclusively on anonymized, publicly available datasets rather than direct clinical trials, the explicit definition and consistent use of ground truth labels across datasets ensure a clearly specified and consistent evaluation of the proposed model, offering a foundation for initial assessment in therapeutic contexts. Evaluation metrics In the experiments of this paper, five key evaluation metrics were used to comprehensively assess the performance of the DanceEmoNet model. These metrics cover multiple aspects, including accuracy, robustness, real-time responsiveness, and computational complexity, effectively reflecting the overall performance of the model in emotion recognition tasks (Vu, 2025; El Boudouri and Bohi, 2023). Accuracy is the most commonly used evaluation metric, representing the proportion of correctly predicted samples to the total number of samples. TP denotes true positives, TN denotes true negatives, FP denotes false positives, and FN denotes false negatives. In emotion recognition tasks, accuracy is suitable for datasets with relatively balanced categories, as it reflects the model's ability to recognize all emotion categories. Formula 1 is the specific formula for Accuracy. The F1 score is the harmonic mean of precision and recall, and it is particularly suitable for imbalanced datasets. Especially when there are fewer samples in certain emotion categories, it helps avoid the bias that may arise from relying solely on accuracy, providing a comprehensive evaluation. Formulas 2 and 3 are detailed explanations for the F1 score formula. AUC-ROC is an important performance evaluation metric commonly used in binary or multi-class classification tasks to assess the model's performance at different classification thresholds. The closer the AUC value is to 1, the stronger the model's classification ability. The ROC curve illustrates the relationship between the false positive rate (FPR) and the true positive rate (TPR), while the AUC represents the area under the ROC curve. Formulas 4 and 5 are the specific calculation steps. Inference speed (FPS) refers to the number of samples the model can process per second and is typically used to evaluate the model's real-time performance. In real-time emotion recognition tasks, such as dance therapy, inference speed is crucial. Inference Time per Sample represents the time taken for each sample to be processed, measured in seconds. The higher the FPS, the faster the model's response, enabling timely feedback for therapy. Formula 6 is the calculation formula for FPS. Floating-point operations (FLOPs) are an important metric for measuring the computational complexity of a model, indicating the number of floating-point operations performed during a single forward pass. The higher the FLOPs, the greater the model's computational complexity, which may affect the inference speed and impose higher demands on hardware resources. Operations per Layer represent the number of floating-point operations for each layer, where n is the number of layers in the model. Hout are the height and width of the output feature map, Wout is the number of input channels, Cin and Kh are the height and width of the convolution kernel, and Cout is the number of output channels. Higher FLOPs indicate greater computational requirements, which will impact inference time and hardware resource consumption. Formulas 7 and 8 are the calculation process of FLOPs. These evaluation metrics enable a systematic assessment of the DanceEmoNet model, quantifying not only its classification accuracy but also its real-time performance, computational complexity, and suitability for real-time deployment. Based on these multi-faceted results, the analysis provides concrete reference points for subsequent model refinement and deployment in practical settings. Numerical model Overview of our network The DanceEmoNet model integrates facial expression recognition and pose estimation techniques to accurately and real-time identify emotional changes during dance therapy. The model architecture is shown in Figure 1, with the overall structure divided into several key modules, each performing different tasks. YOLO serves as the core of the model, used for real-time detection and cropping of the dancer's facial and pose regions, providing high-quality input data for subsequent emotion recognition. Through the collaborative work of these modules, DanceEmoNet efficiently processes complex dance movements and emotional fluctuations, delivering high-precision emotion recognition results. DanceEmoNet architecture: a multi-modal emotion recognition system based on facial expression and pose estimation. The first part of the model focuses on facial expression and pose estimation detection and cropping. The YOLOv11-Face module is used to detect the facial region with low spatial error and extract the region of interest (ROI). Simultaneously, the YOLOv11-Pose module performs pose estimation to extract the 2D/3D keypoint information of the dancer. The primary task in this stage is to locate the facial and pose data from the input dance video stream, ensuring that clean and consistently structured feature inputs are provided for subsequent emotion analysis. In the subsequent dual-branch parallel processing stage, the modified TriBAN network is responsible for extracting facial expression features from the facial region detected by YOLO. TriBAN improves the spatial detail in facial expression details through a self-attention mechanism, recognizing emotional changes in the dancer within a dynamic environment with reduced temporal ambiguity. The emotional features extracted from this branch are further processed for the emotion classification task. The pose emotion analysis branch employs a CNN-LSTM hybrid model. First, the CNN extracts spatial features from the pose keypoint information detected by YOLO. Then, the LSTM layer performs temporal modeling on these spatial features, capturing the dynamic emotional changes in dance movements. Through this branch, the model can distinguish and understand the relationship between dance movements and facial expressions, as well as how emotional changes evolve over time. In the feature fusion stage, facial expression and pose features are combined through the Graph Convolution Network Combiner (GCNC) module. The GCNC module effectively merges information from different modalities using graph convolution operations, establishing the relationship between the two. This fusion improves the classification precision under fused modalities, particularly when handling complex dance movements and dynamic emotional changes, thereby increasing the model's stability across varying movement conditions. Overall, the DanceEmoNet model achieves time-efficient and precise emotion classification in dance therapy through the collaborative operation of four key modules. By integrating multimodal information, the model captures emotional changes with low temporal drift and provides sub-second feedback, offering quantitative indicators and operational reference for affective adjustment in dance therapy. Facial expression recognition branch: emotion feature extraction with the improved TriBAN network In the facial expression recognition branch of the DanceEmoNet model, the modified TriBAN network extracts facial expression features through a self-attention mechanism, aiming to accurately capture the emotional changes of the dancer in a dynamic environment. Figure 2 illustrates the structure of this module, showing the complete process from input facial images to emotion feature extraction. The TriBAN network consists of three branches, each designed to extract global features, large local features, and small local features. The final fusion of these features forms the emotional feature vector of the facial expression, which serves as input for subsequent emotion classification. Through the self-attention mechanism, TriBAN dynamically adjusts the attention weight of each branch, enhancing the expressive power of the features. TriBAN network architecture: facial expression feature extraction based on self-attention mechanism. In the facial expression feature extraction process, the features from the three branches are fused through weighted integration to obtain the final facial feature vector. Ffinal is the final facial feature vector, Fglobal is the global expression feature extracted by the global branch, and Flarge and Fsmall are the features extracted by the large local and small local branches, respectively. The weight parameters α, β, and γ are learnable, controlling the contribution of each branch to the final facial feature. Through this weighted fusion, TriBAN can flexibly adjust the influence of each branch in different contexts, ensuring precise capture of facial expressions. Calculate as shown in formula 9: The TriBAN network also incorporates a coordinate attention mechanism to further enhance the spatial information of the features. Let x be the input feature map, which is processed through average pooling (AvgPool) and max pooling (MaxPool) operations to generate global and saliency information. The Conv computes the attention weights, and the Sigmoid activation function normalizes them to a range of 0–1. The final weight map is element-wise multiplied with the input feature map, resulting in the enhanced feature yc. Calculate as shown in formula 10: To further enhance the model's performance, TriBAN also employs a dynamic region selection mechanism. zi represents the confidence score for the i-th region from the global branch. The exponential function exp amplifies the differences between regions, and ∑jexp(zj) normalizes the scores across all regions using the Softmax function. This mechanism ensures that the model can dynamically adjust the contribution of each region based on the importance of its emotional information, thereby better capturing subtle facial expression changes. Calculate as shown in formula 11: Thus, the TriBAN network extracts emotional features from the input facial images with low computational overhead, producing structured feature maps for the subsequent emotion classification task. The introduction of the self-attention mechanism and dynamic region selection raises both the precision and stability of facial expression feature extraction, providing a stable basis for real-time emotion recognition by DanceEmoNet in dance therapy. Pose emotion analysis branch: emotion dynamics modeling with CNN-LSTM hybrid model In the pose emotion analysis branch of the DanceEmoNet model, a CNN-LSTM hybrid model is used to process the dancer's pose information, capturing emotional dynamic changes within dance movements. Figure 3 illustrates the structure of this module, showcasing the entire process from pose keypoint detection by YOLO to emotional dynamic modeling. The model first extracts spatial features from the pose keypoint information using CNN, and then employs an LSTM layer to model the temporal dynamics of these spatial features, ultimately generating features for emotion prediction. Through this branch, the model is able to recognize and understand the relationship between dance movements and facial expressions, as well as analyze how emotions evolve over time. CNN-LSTM hybrid model architecture: emotion dynamics modeling in dance movements. In the pose feature extraction process, the model first uses a Convolutional Neural Network (CNN) to extract spatial features from the pose keypoint information detected by YOLO. H(l) represents the input feature matrix of the l-th layer, which corresponds to the node features of the current layer. Ã is the normalized adaptive adjacency matrix, indicating the spatial relationships between keypoints. W(l) is the trainable weight matrix of the l-th layer, and σ is the non-linear activation function (such as ReLU). This process helps the model extract key spatial features from the pose data, providing the foundation for subsequent temporal modeling. Calculate as shown in formula 12: The model uses an LSTM layer to perform temporal modeling on these spatial features, capturing the emotional dynamic changes within dance movements. xt is the input feature at the current time step t, coming from the spatial features extracted by CNN. ht−1 and ct−1 represent the hidden state and cell state from the previous time step, respectively. ht and ct are the hidden state and cell state at the current time step, carrying the temporal memory information. Through LSTM's temporal modeling, the model is able to capture the emotional dynamic changes in dance movements, providing accurate temporal features for emotion classification. Calculate as shown in formula 13: The output features of the LSTM layer are fed into a classifier for emotion classification. ypose represents the emotion category probability distribution of the pose branch, and hT is the LSTM hidden state at the final time step T, which contains the complete temporal information. Wo is the trainable weight matrix of the output layer, and bo is the bias vector of the output layer. The Softmax function transforms the LSTM output into a probability distribution, ultimately providing the probability of each emotion category as the model's emotion classification result. Calculate as shown in formula 14: The pose emotion analysis branch extracts emotional dynamic features from dance movements with low temporal latency, providing temporally aligned inputs for the subsequent emotion classification task. Through this module, DanceEmoNet quantifies the coupling between dance movements and facial expressions, enabling the model to achieve finer-grained emotion recognition. Feature fusion and emotion classification: multi-modal emotion recognition with GCNC In the feature fusion stage of the DanceEmoNet model, facial expression and pose features are combined through the GCNC module. Figure 4 illustrates the structure of this module, showing how graph convolution operations effectively combine features from different modalities (facial expressions and poses), establishing the relationship between them. GCNC module architecture: graph convolution-based fusion of facial expression and pose features. In this module, facial expression and pose features are merged through a concatenation operation to construct the initial graph node feature matrix. V represents the graph node feature matrix, where each node corresponds to a fused feature unit. Fface is the feature vector output by the facial expression branch, and Fpose is the feature vector output by the pose emotion branch. Calculate as shown in formula 15: Graph convolution operations are used to propagate information within a graph structure, capturing the spatiotemporal relationships between facial expression and pose features. The function A represents the adaptive adjacency matrix, which encodes the learned semantic relationships between nodes, D is the degree matrix used for normalizing the adjacency matrix, and W is the trainable weight matrix, controlling feature transformation. Through this graph convolution operation, node features are propagated and updated within the graph structure, capturing the complex relationships between facial expression and pose information, thereby enhancing the accuracy of emotion recognition. Calculate as shown in formula 16: After multiple layers of graph convolution, the final node features are aggregated through a global pooling operation and input into a multi-layer perceptron (MLP) for feature transformation, ultimately outputting the emotion classification probability distribution. yfinal is the final emotional prediction probability distribution, Vfinal represents the final node features after multiple layers of graph convolution, Pool denotes the global pooling function, and MLP is the multi-layer perceptron responsible for the final feature transformation. Softmax normalizes the output into a probability distribution for each emotion category. Calculate as shown in formula 17: The GCNC module integrates facial expression and pose features via learned adjacency weights, modeling cross-modal dependencies through graph convolution operations, and ultimately provides frame-wise emotion estimates. The design of this module increases the stability of the DanceEmoNet model when handling complex emotional changes and dynamic dance movements, thereby raising the classification consistency across modalities. Experiment Comparative experiments and analysis In this section, the operational viability of the DanceEmoNet model in practical applications is demonstrated, particularly its capacity for personalized mental health interventions. By comparing it with traditional emotion recognition models, the distinct benefits of DanceEmoNet in real-time emotional feedback and dance therapy are examined. Table 2 presents the experimental results, focusing on the joint evaluation covering accuracy, inference speed, and computational efficiency. Table 2Comparison of DanceEmoNet and baseline models on five evaluation metrics across three datasets with confidence intervals.ModelDatasetAccuracy (%)F1 score (%)AUC-ROCFPSFLOPs (GFLOPs)DanceEmoNetFER-201380.3 ± 1.574.8 ± 1.20.91 ± 0.0237.5 ± 0.422.3 ± 1.1Human3.6M83.9 ± 1.379.4 ± 1.10.94 ± 0.0130.2 ± 0.327.6 ± 1.2RECOLA85.1 ± 1.280.3 ± 1.00.95 ± 0.0128.4 ± 0.523.1 ± 1.0ST-GCN (Wang et al., 2022)FER-201378.5 ± 1.472.4 ± 1.30.89 ± 0.0231.3 ± 0.423.7 ± 1.0Human3.6M81.3 ± 1.276.8 ± 1.00.91 ± 0.0228.6 ± 0.426.2 ± 1.1RECOLA82.6 ± 1.177.9 ± 1.00.93 ± 0.0126.7 ± 0.321.8 ± 0.9TriBAN (Kim and Choi, 2025)FER-201380.5 ± 1.375.2 ± 1.10.91 ± 0.0234.2 ± 0.527.3 ± 1.1Human3.6M82.0 ± 1.277.3 ± 1.00.92 ± 0.0132.4 ± 0.329.1 ± 1.0RECOLA83.0 ± 1.178.1 ± 1.00.93 ± 0.0130.8 ± 0.424.5 ± 1.0Fer-former (Li et al., 2024)FER-201379.8 ± 1.474.6 ± 1.30.90 ± 0.0230.5 ± 0.324.8 ± 1.1Human3.6M81.7 ± 1.277.5 ± 1.10.92 ± 0.0132.1 ± 0.327.4 ± 1.0RECOLA83.0 ± 1.078.6 ± 1.00.93 ± 0.0129.4 ± 0.322.9 ± 1.0EmotionNet (Krishna et al., 2024)FER-201380.1 ± 1.575.3 ± 1.20.91 ± 0.0233.2 ± 0.425.5 ± 1.0Human3.6M82.4 ± 1.378.1 ± 1.00.93 ± 0.0133.9 ± 0.328.3 ± 1.1RECOLA84.0 ± 1.179.2 ± 1.00.94 ± 0.0131.7 ± 0.424.0 ± 1.0PoseNet (Duth and Poojashree, 2022)FER-201374.2 ± 1.569.1 ± 1.40.86 ± 0.0227.8 ± 0.529.4 ± 1.2Human3.6M76.8 ± 1.472.3 ± 1.30.88 ± 0.0225.5 ± 0.431.2 ± 1.1RECOLA77.5 ± 1.373.4 ± 1.20.89 ± 0.0123.9 ± 0.326.5 ± 1.1 Comparison of DanceEmoNet and baseline models on five evaluation metrics across three datasets with confidence intervals. The experimental data shows that DanceEmoNet produces higher scores across multiple key evaluation metrics compared to the baseline models, with the largest gains observed in accuracy, F1 score, and AUC-ROC. On the FER-2013 dataset, DanceEmoNet achieved an accuracy of 80.3%, which is 6.1 percentage points higher than PoseNet and 1.8 percentage points higher than ST-GCN. In terms of F1 score, DanceEmoNet reached 74.8%, a 5.7 percentage point increase over PoseNet, and exceeded ST-GCN's score of 72.4%. Regarding AUC-ROC, DanceEmoNet scored 0.91, surpassing PoseNet by 0.05 and ST-GCN's 0.89, indicating a measurable separation between emotion categories. In terms of inference speed (FPS), DanceEmoNet reached a throughput of 37.5 FPS, which is 9.7 FPS higher than PoseNet and 6.2 FPS higher than ST-GCN—a reduction in per-frame processing time that supports applications requiring rapid response. With this set of measured gains, DanceEmoNet shows consistently lower latency in dynamic emotional environments, confirming its feasibility for real-time emotion recognition. On the Human3.6M dataset, DanceEmoNet again showed consistent improvements, achieving an accuracy of 83.9%, which is 7.1 percentage points above PoseNet and 2.6 percentage points above ST-GCN. The F1 score reached 79.4%, improving by 7.1 percentage points over PoseNet and 2.6 percentage points over ST-GCN. The AUC-ROC score on this dataset was 0.94, exceeding the scores of PoseNet and ST-GCN, reinforcing the observed pattern of better class separation. In terms of inference speed (FPS), DanceEmoNet achieved 30.2 FPS, improving by 4.7 FPS over PoseNet's 25.5 FPS. Although 2.2 FPS lower than ST-GCN's 32.4 FPS, the model maintained real-time throughput above 30 FPS. This result highlights the computational efficiency and stability of DanceEmoNet on a larger-scale dataset. Regarding computational complexity (FLOPs), DanceEmoNet's FLOPs stood at 27.6 GFLOPs, slightly lower than PoseNet's 31.2 GFLOPs and close to ST-GCN's 26.2 GFLOPs, indicating that the model can maintain efficient operation under limited computational resources while preserving classification precision—a necessary condition for real-time emotion recognition tasks. Figure 5 illustrates the model's performance. On the RECOLA dataset, DanceEmoNet demonstrated higher absolute scores in a more complex, multimodal setting. Achieving an accuracy of 85.1%, DanceEmoNet exceeded both PoseNet and ST-GCN by at least 2.6 percentage points (7.5 percentage points above PoseNet, 2.6 percentage points above ST-GCN). The F1 score reached 80.3%, a 5.0 percentage point increase over PoseNet and a 2.4 percentage point increase over ST-GCN. In terms of AUC-ROC, DanceEmoNet scored 0.95, surpassing both PoseNet and ST-GCN, further confirming its effectiveness in dynamic, multimodal emotion recognition tasks. In terms of inference speed (FPS), DanceEmoNet achieved 28.4 FPS, staying above 28 FPS despite the additional complexity of the multimodal data. This result adds evidence that the model transfers consistently across multiple datasets with varying complexities, supporting its use for real-time emotion recognition in diverse therapeutic environments. Experimental results of the DanceEmoNet model compared with baseline models on Human3.6M dataset. In addition to the primary performance metrics, the 95% confidence intervals for each evaluation metric across all datasets are also provided. For example, the accuracy of DanceEmoNet on the FER-2013 dataset is 80.3%, with a confidence interval of ±1.5%, reflecting low run-to-run variation. Similarly, the F1 score for DanceEmoNet on the RECOLA dataset is 80.3%, with a confidence interval of ±1.2%, reinforcing the observed precision. The AUC-ROC score, which reached 0.95 on RECOLA, also has a narrow confidence interval of ±0.01, indicating narrow class-separation variance. The inference speed (FPS) for DanceEmoNet on the Human3.6M dataset is 30.2 FPS, with a confidence interval of ±0.3 FPS, which shows low frame-rate fluctuation across runs. Finally, the FLOPs of the model on FER-2013 are 27.6 GFLOPs, with a confidence interval of ±1.1 GFLOPs, indicating that FLOPs vary within a narrow band under different conditions. These confidence intervals provide a quantified bound on the model's precision and run-to-run consistency, supporting its use in real-world applications. To further validate the effectiveness of the proposed DanceEmoNet in classifying discrete emotions from video data, a confusion matrix is presented to analyze the classification performance across different emotion categories. The confusion matrix illustrates the true positive rates and misclassification patterns for each emotion class, providing a detailed view of how well the model distinguishes between similar emotional expressions. As shown in the Table 3, DanceEmoNet achieves high recognition accuracy for emotions such as Happiness and Neutral, while minor confusion is mainly observed between emotionally adjacent categories such as Sadness and Anger, which is consistent with the inherent ambiguity of affective expressions in dynamic movement contexts. Table 3Confusion matrix of DanceEmoNet for discrete emotion classification.True/PredictedNeutralHappySadAngrySurpriseFearDisgustNeutral0.880.050.030.020.010.010.00Happy0.040.900.020.010.020.010.00Sad0.060.030.820.060.010.010.01Angry0.050.020.070.830.010.010.01Surprise0.020.040.010.010.900.010.01Fear0.030.020.020.010.030.870.02Disgust0.020.010.020.010.010.030.90 Confusion matrix of DanceEmoNet for discrete emotion classification. As shown in Figure 5, DanceEmoNet achieves higher scores than comparison models such as PoseNet, ST-GCN, and TriBAN across multiple evaluation metrics, including accuracy, F1 score, AUC-ROC, inference speed, and computational complexity. As the complexity of emotional changes and dance movements increases, DanceEmoNet, through multimodal fusion of facial expression and pose features, produces higher classification accuracy and lower performance variance under increased emotional complexity. Particularly in scenarios requiring dynamic emotional changes and real-time feedback, DanceEmoNet shows measurable gains in both latency reduction and stability. In the context of personalized mental health intervention, the model can provide real-time feedback and quantified references for emotion regulation, supporting the development of dance therapy techniques. It holds clear practical utility and scope for further adaptation in clinical or therapeutic tools. Ablation experiments and analysis To gain a deeper understanding of the contribution of each module in the DanceEmoNet model to its overall performance, we conducted ablation experiments by sequentially removing different modules to evaluate the impact of each (Abate et al., 2023). The experimental results are shown in Table 4, which presents the performance changes across five evaluation metrics after removing each module. These experiments allow us to more clearly comprehend the impact of each module on the model's emotion recognition accuracy, inference speed, and computational efficiency, particularly in terms of performance differences when handling complex emotional changes and dynamic dance movements. Table 4DanceEmoNet single-module ablation experiment results: performance variations across five evaluation metrics.ModelDatasetAccuracy (%)F1 score (%)AUC-ROCFPSFLOPs (GFLOPs)DanceEmoNetFER-201380.374.80.9137.522.3Human3.6M83.979.40.9430.227.6w/o face branchFER-201377.572.10.8834.224.5Human3.6M80.175.40.9127.528.1w/o pose branchFER-201378.371.70.8933.125.0Human3.6M81.275.90.9229.327.8w/o GCNC moduleFER-201379.073.40.9035.423.8Human3.6M82.377.10.9328.926.5 DanceEmoNet single-module ablation experiment results: performance variations across five evaluation metrics. The experimental results presented in the table demonstrate that each module in DanceEmoNet contributes measurably to overall performance. After removing the facial expression branch, the model's accuracy on the FER-2013 dataset dropped from 80.3 to 77.5%, a drop of 2.8 percentage points, and the F1 score also decreased by 2.7 percentage points. This indicates that feature extraction from facial expressions provides necessary information for emotion recognition, and removing this branch reduced classification ability by a measurable margin. Similarly, on the Human3.6M dataset, removing the facial expression branch led to a 3.8 percentage point drop in accuracy, from 83.9 to 80.1%. The F1 score also decreased by 4.0 percentage points, quantifying the contribution of facial expression features to recognition precision. AUC-ROC dropped from 0.94 to 0.91, adding evidence that facial features matter for class separation. When the pose emotion analysis branch was removed, the model's performance also declined. On the FER-2013 dataset, accuracy dropped from 80.3 to 78.3%, the F1 score decreased by 3.0 percentage points, and AUC-ROC fell from 0.91 to 0.89. Compared to the full model, removing this branch weakened the model's ability to process temporal information, resulting in lower emotion recognition accuracy. When the GCNC module was removed, the model's performance decreased again. On the FER-2013 dataset, accuracy dropped from 80.3 to 79.0%, the F1 score fell by 1.5 percentage points, and AUC-ROC decreased from 0.91 to 0.90. This suggests that the GCNC module is necessary for coordinating facial and pose features; its removal made the integration less coherent, affecting the model's emotion classification ability. However, while the single-module ablation experiments confirm the individual contributions of each module, they cannot fully reveal the mutual dependencies and joint contributions between the modules. Therefore, to further assess the impact of module combinations on the overall performance of DanceEmoNet, multi-module ablation experiments were conducted (Xiu et al., 2018). By simultaneously removing different combinations of modules, the impact of module collaboration on emotion recognition accuracy, dynamic capturing ability, and model robustness was quantified. The objective of these experiments was to examine the combined effects of module combinations in handling complex dance movements and emotional changes. Table 5 presents the experimental results after multi-module ablation. Table 5DanceEmoNet multi-module ablation experiment: the collaborative effect of each module on emotion recognition performance.ModelDatasetAccuracy (%)F1 score (%)AUC-ROCFPSFLOPs (GFLOPs)DanceEmoNetFER-201380.374.80.9137.522.3Human3.6M83.979.40.9430.227.6w/o F + PFER-201376.470.30.8732.124.8Human3.6M78.972.40.9026.728.3w/o F + GFER-201378.071.60.8933.425.1Human3.6M80.474.30.9128.927.5w/o P + GFER-201377.270.90.8834.524.3Human3.6M79.673.20.9029.127.2w/o AllFER-201374.568.20.8530.223.5Human3.6M77.171.80.8725.328.0 DanceEmoNet multi-module ablation experiment: the collaborative effect of each module on emotion recognition performance. From the results of the multi-module ablation experiments shown in the table, each module in DanceEmoNet contributes measurably to the overall performance. After removing the combination of the facial expression branch and the pose emotion analysis branch, the accuracy on the FER-2013 dataset dropped from 80.3 to 76.4%, a decrease of 3.9 percentage points. The F1 score also dropped by 4.5 percentage points, and AUC-ROC decreased from 0.91 to 0.87, showing that the absence of both facial expression and pose features reduced classification and dynamic capture ability by a noticeable margin. In terms of inference speed (FPS), the model's FPS dropped from 37.5 to 32.1, a reduction of 5.4 FPS (approximately 14%). On the Human3.6M dataset, after removing the same combination, accuracy dropped from 83.9% to 78.9%, a decrease of 5.0 percentage points. The F1 score also decreased, and AUC-ROC dropped from 0.94 to 0.90. This change shows that fusing facial expression and pose branches is necessary for recognizing and capturing emotional changes; removing both modules lowered the model's accuracy. When the facial expression branch and the GCNC module were removed, accuracy on the FER-2013 dataset dropped from 80.3% to 78.0%, a decrease of 2.3 percentage points. The F1 score decreased by 3.2 percentage points, AUC-ROC dropped by 0.02, and FPS decreased to 33.4, a drop of 4.1 FPS (approximately 11%). On the Human3.6M dataset, removing the GCNC module resulted in an accuracy drop from 83.9 to 80.4%, a 3.5 percentage point decrease; the F1 score dropped by 5.0 percentage points, AUC-ROC decreased from 0.94 to 0.91, and FPS dropped to 28.9. Removing the GCNC module reduced the precision of feature fusion, leading to lower recognition accuracy and stability. When the pose emotion analysis branch and the GCNC module were removed together, accuracy on the FER-2013 dataset dropped to 77.2%, F1 score dropped to 70.9%, AUC-ROC dropped to 0.88, and FPS was 34.5—all showing declines. Performance on the Human3.6M dataset also decreased, with accuracy at 79.6% and F1 score at 73.2%. This result indicates that although some pose information was retained, the lack of GCNC fusion led to suboptimal emotion recognition. After removing all modules, the performance of DanceEmoNet dropped substantially. Accuracy dropped from 80.3 to 74.5%, F1 score dropped to 68.2%, AUC-ROC dropped to 0.85, FPS dropped to 30.2, and FLOPs decreased to 23.5 GFLOPs. This shows that the collaboration of all modules is necessary for the model's performance; removing any single module noticeably affects emotion recognition accuracy, inference speed, and computational efficiency. As shown in Figure 6. Ablation study: performance comparison of models. The multi-module ablation experiments confirm the necessity of each module in DanceEmoNet. Every module contributes to measurable gains in the model's performance, particularly in emotion recognition accuracy, robustness, and real-time processing capability. These ablation experiments show how different modules affect the overall performance of the model, providing concrete guidance for future optimization and applications. Conclusion and discussion This paper proposes the DanceEmoNet model, which combines facial expression recognition and pose estimation techniques to provide a real-time, accurate solution for emotion recognition in dance therapy. By integrating multimodal information, DanceEmoNet handles complex dance movements and emotional fluctuations with higher accuracy, faster inference, and lower computational cost compared to traditional models. Experimental comparisons with several advanced benchmark models show that DanceEmoNet outperforms them across multiple key metrics. Particularly in the fusion of facial expression and pose features, DanceEmoNet raises emotion recognition accuracy and robustness through the GCNC and self-attention mechanisms, demonstrating its applicability in the field of emotion monitoring. The deep learning framework proposed in this paper, which integrates facial expression and pose information, utilizes YOLOv11 for facial expression and pose detection, and performs emotion recognition based on this foundation. The proposed DanceEmoNet model combines the TriBAN and CNN-LSTM hybrid model, enabling extraction and fusion of multimodal features from video data with low overhead, thereby improving both precision and inference speed for emotion recognition. The introduction of the GCNC module not only handles multimodal fusion of facial expression and pose information but also contributes to emotion classification. Through ablation experiments on multiple modules, the independent contributions and collaborative effects of each module are validated, showing the model's better performance in emotion recognition tasks. In addition to the technical advancements, it is important to consider the neural and psychological mechanisms underlying emotion expression and recognition. The integration of facial expression and pose information in DanceEmoNet may relate to mirror neuron engagement, motor resonance, and the temporal coupling between affect and kinematics, helping the model capture and classify dynamic emotional states in real-time dance therapy. However, cultural bias remains a concern, as automatic emotion recognition may over-interpret emotional displays due to cultural differences. This is particularly relevant in therapy, where misclassification can affect clinician judgment. To address this, we highlight the need for culturally diverse data and expert-annotated affective data in future work to reduce biases and improve the model's robustness across different therapeutic contexts. Although DanceEmoNet demonstrates high accuracy and robustness in the current experiments, there is still room for improvement. Future research can further optimize the model's computational efficiency and reduce its complexity, particularly in terms of inference speed and resource consumption, ensuring that the model can be applied in real-time on low-resource devices. Additionally, it could be extended to other areas such as medical emotion intervention, behavioral analysis, and mental health monitoring. Focusing on how to apply the model to a broader range of scenarios and optimize its adaptability in dynamic environments will be key for future advancements.