An effective two-stage key frame extraction method for speech-visual emotion recognition Authors: Yuanhui Liang, Xiaojing Li, Lei Wang Published: 2026 ABSTRACT AbstractSpeech-visual emotion recognition plays a vital role in human–computer interaction applications. However, it typically confronts several challenges, such as: (1) conventional speech-visual key frame (SVKF) extraction methods are susceptible to redundancy and emotional information loss; (2) widely adopted attention-based speech-visual feature fusion approaches often compute weights with limited interpretability. To address these challenges, this paper proposes an effective two-stage key frame extraction method for speech-visual emotion recognition. Specifically, in the first stage, visual key frames (VKFs) are extracted by employing information entropy (IE) to model the continuous process of emotion generation, thereby decreasing visual frame redundancy. Corresponding speech key frames (SKFs) are obtained simultaneously by eliminating silent segments to reduce redundancy in the speech modality. Subsequently, by leveraging the complementarity characteristics of speech and visual modalities, the first-stage SKFs and VKFs are aligned to produce the ultimate second-stage SVKFs for preserving important emotional information, in which a simple and interpretable weighted fusion is also proposed to focus on processing important emotional information. The experimental results on the RML, eNTERFACE05, MEAD and BAUM-1s datasets demonstrate that the proposed effective two-stage key frame extraction has better inference and generalization performance.Keywords: Emotion recognition, Speech modality, Visual modality, Key framesSubject terms: Computational biology and bioinformatics, Mathematics and computing, Neuroscience Introduction The purpose of speech-visual emotion recognition is to identify human emotions by utilizing some intelligent devices1,2. As the core component of human-computer interaction systems3,4, speech-visual emotion recognition has significant potential to be applied to intelligent driving5, smart home6, health detection7, smart education8, and so on. Despite the great application promise of speech-visual emotion recognition, it is still a challenge to recognize emotions effectively9. Especially against the backdrop of the rapid proliferation of various social media application platforms, which have generated vast amounts of emotional videos10, the quality of speech-visual key frames (SVKFs) extracted from these large-scale video datasets directly influences the performance of emotion recognition systems. Therefore, how to effectively extract high-quality SVKFs from massive emotional video data has become a critical research priority. On the other hand, the effective fusion of speech-visual features constitutes another crucial task in speech-visual emotion recognition11,12. Attention-based feature fusion methods have been widely adopted in this field, as they integrate speech and visual features through an adaptive weighting mechanism13–15. However, such methods generally require incorporating additional neural network layers to learn the attention weights, which can inadvertently increase model complexity. Moreover, since these attention weights are implicitly derived by training the appended networks on large-scale datasets, the resulting fusion process often suffers from limited interpretability16. Based on the motivations outlined above, this paper is driven by two primary objectives. The first is to efficiently extract SVKFs that exhibit low redundancy while preserving important emotional information. The second is to enhance the interpretability of the weighted fusion mechanism for speech and visual features. To achieve these goals, this paper proposes an effective two-stage key frame extraction method for speech-visual emotion recognition. The main contributions of this work are summarized as follows: The first-stage visual key frame (VKF) extraction algorithm is proposed to reduce the redundancy of key frames. The first-stage visual key frame (VKF) extraction algorithm is proposed to reduce the redundancy of key frames. The first-stage SVKF alignment algorithm is proposed to preserve important emotional information. The first-stage SVKF alignment algorithm is proposed to preserve important emotional information. A simple yet interpretable weighted fusion mechanism is designed to emphasize important emotional information. A simple yet interpretable weighted fusion mechanism is designed to emphasize important emotional information. Experimental results on four public speech-visual emotion recognition datasets show that the proposed effective two-stage key frame extraction has better inference and generalization performance. Experimental results on four public speech-visual emotion recognition datasets show that the proposed effective two-stage key frame extraction has better inference and generalization performance. Related work The topic of this paper is relative to the SVKE extraction and the speech-visual feature fusion. In the following sections, the related work on these two topics is described in detail. SVKF extraction Several SVKF extraction algorithms have been investigated for speech-visual emotion recognition. In particular, in one approach, emotional videos were separated into speech and visual information, where the visual frames with the smallest chi-square distance of seven adjacent frames for the visual information were used as the VKFs17. A limitation of this method, however, is that the speech information remained unprocessed, leading to redundancy in the extracted speech key frames (SKFs). Similarly, another study divided the speech information into silent and speech segments, treating the latter as SKFs, while the corresponding visual frames aligned with these SKFs were regarded as VKFs18. Although this strategy reduces the redundancy of SVKFs, it will lose some important emotional information. For example, in one extreme case, people are particularly surprised and may be unable to speak, i.e., corresponding speech information is the silent segments, while people’s facial expressions provide more emotional information. In this case, the corresponding speech-visual information is inevitably excluded, since the corresponding speech information is the silent segments. This causes the loss of important emotional information, as facial expressions that provide more emotional information are discarded. In another work, VKFs were obtained using fixed frame intervals for the visual information, while SKFs were derived by applying endpoint detection to segment the speech information into equidistant segments19. Other methods involved directly splitting emotional videos into speech-visual frame pairs, which were then fed into multimodal feature extraction networks14,20–23. It is remarkable that SVKFs extracted by the above methods generally suffer from key frame redundancy with missing important emotional information. For this reason, this paper proposed an effective two-stage key frame extraction method for speech-visual emotion recognition. It first acquires the first-stage SKFs and VKFs to reduce redundancy, and then saves the important emotional information by utilizing the complementarity characteristics of speech-visual modalities to align the first-stage SKFs and VKFs. Speech-visual feature fusion Unimodal speech or visual emotion recognition provides insufficient emotional information, which is a major limitation for emotion recognition22–26. On the other hand, different emotion modalities generally contain different dimensions of emotional information, in which these modalities can complement each other to provide a more comprehensive representation information27. Consequently, the fusion of speech-visual features is crucial for improving the performance of speech-visual emotion recognition. It is currently more popular to use different neural networks to fuse the extracted speech-visual features11. Typically, speech and visual features have been extracted using two-dimensional convolutional neural networks (2D-CNNs)28 and three-dimensional CNNs (3D-CNNs)29, respectively, and subsequently fused via a deep belief network (DBN)12. Likewise, multi-layer perceptrons (MLPs) have also been employed to integrate speech-visual features30. In addition, the concatenation operator31,32 has been frequently used to fuse speech-visual features. Building on this, a novel speech-visual loss function (NSVLF)33 has been designed to enhance the robustness of feature fusion. Based on speech-visual correlations, a multi-stage adaptive feature fusion (MSAFF) method34 has been proposed to dynamically adjust speech-visual informatiion. Since the attention mechanism enables adaptive weighting of different modalities, it has also been applied to the fusion of speech-visual features. An importance attention and a complementary attention were designed to achieve the weighting of different emotion modalities13. Further extending this line of work, an adaptive attention mechanism was proposed to dynamically weight speech-visual modalities14. Although these weighting methods based on the attention mechanism could improve the performance of speech-visual emotion recognition, their attention weights need to be learned by adding neural network layers. This potentially introduces additional complexity. Further, the attention weights have relatively low interpretability, as it is learned implicitly by training additional neural network layers on a large amount of data16. To this end, this paper proposes a simple and interpretable weighted fusion method to dynamically weight the extracted speech-visual features, in which the weights are explicitly computed by the proposed effective two-stage key frame extraction based on the amount of important emotional information. Proposed effective two-stage key frame extraction method The quality of key frames directly affects the performance of speech-visual emotion recognition. Nevertheless, current key frame extraction methods for speech-visual emotion recognition confront the concerns of key frame redundancy and dropping important emotional information. To this end, this work proposes an effective two-stage key frame extraction method, illustrated in Fig. 1. Specifically, an emotional video is first decomposed into speech frames and visual frames , where N, and indicate the total number of frames, the i-th speech and visual frames, respectively. Subsequently, by leveraging information entropy (IE) to model the continuous process of emotion generation to reduce the redundancy of , the first-stage VKFs with corresponding position parameters are derived using the proposed Algorithm 1. Meanwhile, to alleviate the redundancy in , the first-stage SKFs with corresponding position parameters are obtained by using voice activity detection (VAD). Finally, to save important emotional information, the second-stage SVKFs and are extracted by utilizing the proposed Algorithm 2 to align and based on the complementarity characteristics of speech-visual modalities, in which the interpretable speech-visual weights and are also calculated. The framework of the proposed effective two-stage key frame extraction method. First-stage VKFs The generation of emotion is a continuous process, i.e., it generally goes through the following steps: beginning, maximizing, reducing, and ending35. If the visual frames corresponding to the maximum step are extracted, the redundancy of is effectively reduced. On the other hand, a visual image can be viewed as a two-dimensional discrete signal, where the information of the visual image can be computed by the information entropy (IE)36. Further, when the IE of the visual image is higher, it better reflects the main information of the visual image, and vice versa37. Accordingly, to reduce the redundancy of , this paper extracts the first-stage VKFs by leveraging the IE to model the continuous process of emotion generation. Details are as follows. Firstly, the IE of is calculated. To be specific, let represent the size of , and represent the number of gray level j for , where is the coordinate. As a result, the probability of the gray level j for is defined as Based on Eq.(1), the IE of is defined as where represent the gray level range of j for . According to Eq.(2), the average IE of is defined as Based on Eq.(2), the IE of each frame in is calculated. Since the generation of emotion is continuous, we first define the visual frames corresponding to the maximum value of the IE as the visual reference frames , in which the set of the position parameter i of is defined as . Meanwhile, to ensure that are all valid, the IE of should be greater than or equal to in Eq.(3). Secondly, based on and , the first-stage VKFs are extracted. Concretely, from the definition of , it can be seen that the emotional information contained in is higher than other visual frames, which is used as a part of the first-stage VKFs. However, the number of usually do not meet the requirement of feature extraction networks. For this reason, it is necessary to select other visual frames as a part of the first-stage VKFs. It is widely known that comparing similarity between visual frames is an effective approach to meet the above needs. Consequently, this paper calculates the similarity between and other visual frames to determine whether other visual frames are used as the first-stage VKFs. In calculating the similarity, the perceptual hash algorithm38 with the Hamming distance39 is applied, since it is a simple and computationally fast similarity comparison method. Subsequent, based on the similarity calculation results, and the visual frames similar to are defined as the first-stage VKFs , where the set of the position parameter i of is defined as . For example, for a 5-frame visual information , if frames 2 and 4 are the first-stage VKFs, then and . Finally, the first-stage VKF extraction is summarized in Algorithm 1. First-Stage VKF Extraction First-stage SKFs Speech information can be divided into speech and silent segments, where the emotional features of the silent segments are calculated to close to zero18. That is, the silent segments do not improve the performance of emotion recognition, but it will cause data pollution to reduce the performance of emotion recognition. For this reason, the VAD18 algorithm can be applied to eliminate silent segments to reduce the redundancy of , where the remaining speech information is defined as the first-stage SKFs . Meanwhile, the set of the position parameter i for is defined as . For example, for a 5-frame speech information , if frames 2 and 5 are the first-stage SKFs, then and . First-stage SVKF alignment For and , based on “First-stage VKFs” and “First-stage SKFs”, the first-stage SVKFs with corresponding position parameters are obtained, i.e., , , and . Although and effectively reduce the redundancy of and , it is difficult to save some important emotional information. For this purpose, and are aligned according to the time steps (i.e., the position parameters in and ), which includes following three cases. Both the first-stage SVKFs correspond each other at these time steps, and thus information from both streams can be considered. Both the first-stage SVKFs correspond each other at these time steps, and thus information from both streams can be considered. Only the first-stage VKFs are available at these time steps as the speech contains of silent segments. Only the first-stage VKFs are available at these time steps as the speech contains of silent segments. Only the first-stage SKFs are available at these time steps as the visual frames with less emotional information are discarded by using the IE. Only the first-stage SKFs are available at these time steps as the visual frames with less emotional information are discarded by using the IE. For the above three cases, traditional SVKF extraction algorithms usually save the speech-visual frames that belong to the case (1), while the speech-visual frames that belong to the case (2) and case (3) are directly discarded. This results in the loss of important emotional information. The reasons are as follows: The case (2) should be processed with emphasis. For example, in one extreme case, people are too surprised to speak (corresponding speech frame is a silent segment), but people’s facial expressions provide more emotional information (corresponding visual frame is the first-stage VKFs). For such extreme case, the visual frame provides more emotional information that is particularly important for speech-visual emotion recognition, which should be processed with emphasis. Similarly, the speech-visual frames corresponding to the case (3) should not be discarded, which is also processed with emphasis. Thus, to save the important emotional information of the case (2) and case (3) in and , this paper applies the complementary characteristics between speech and visual modalities to effectively extract the second-stage SVKFs, in which the handing of each case is shown in Fig. 2 (To make it easier to understand how to obtain the second-stage SVKFs, a simple example can be found in Appendix 6). The handling of each case. For the case (1), both the first-stage SVKFs correspond each other at these time steps, which are used directly as a part of the second-stage SVKFs. Meanwhile, the set of position parameter for the speech-visual frames that belong to the case (1) are defined as and , respectively. For the case (2), it should include extreme cases (e.g., the particularly surprising example mentioned above) and normal cases. For the extreme cases, it corresponds to a higher IE. To improve the performance of speech-visual emotion recognition, the visual frames corresponding to the extreme cases should be retained, where the speech frames corresponding to the extreme cases should also be supplemented. This is because speech and visual modalities are complementary properties20. That is, if the visual frames corresponding to the extreme cases provide more emotional information, then the speech frames corresponding to the extreme cases should also usually provide more emotional information. For the normal cases, it corresponds to a lower IE, where the speech and visual frames corresponding to the normal cases can be discarded. Consequently, to distinguish the normal and extreme cases in the case (2), this paper calculates the average IE of , which is defined as where represents the length of the vector. is the index of . is the position parameter corresponding to the a-th index. is calculated by Eq.(2), e.g., if , then and . Meanwhile, let represent the number of frames belonging to the case (2) in an emotional video of N frames, and let represent the IE of the -th visual frame belonging to the case (2). If , then can be regarded as the extreme cases, where the position parameters are saved in the set , otherwise can be viewed as the normal cases and is discarded. In addition, the speech frames corresponding to the extreme cases are complemented. Specifically, this work first finds the visual frames in with the IE closest to the extreme cases, where the position parameters of the visual frames in are defined as , and then the speech frames are used as the complemented speech frames corresponding to the extreme cases. In summary, the complemented speech frames of the extreme cases and the visual frames corresponding to the extreme cases are defined as a part of the second-stage SVKFs. Similarly, for the case (3), it should include the extreme and normal cases. The speech frames corresponding to the extreme cases should be saved, where the visual frames corresponding to the extreme cases should also be supplemented. In addition, the speech frames corresponding to the normal cases should be discarded. On the other hand, the high-energy speech frames generally contain more emotional information18. Thus, this paper applies the speech energy to distinguish the normal and extreme cases in the case (3), in which the average energy of is defined as where is the index of . is the position parameter corresponding to the b-th index. is the -th speech frame, e.g., if , then and . In addition, let represent the number of frames belonging to the case (3) in an emotional video of N frames, and let represent the energy value of the speech frame belonging to the case (3). If , then can be considered as the extreme cases and the position parameters are saved in the set , otherwise is viewed as the normal cases and is discarded. Likewise, this paper also complements the visual frames corresponding to the extreme cases. To be specific, this work first finds the speech frames in with energy value closest to the extreme cases, where the position parameters of the speech frames in are defined as , and then the visual frames are used as the complemented visual frames corresponding to the extreme cases. In summary, the complemented visual frames of the extreme cases and the speech frames corresponding to the extreme cases are defined as a part of the second-stage SVKFs. Combining the case (1), case (2) and case (3), the ultimate second-stage SVKFs for and are defined as Since the extreme cases contain important emotional information, which are set with large weights to effectively improve the accuracy of speech-visual emotion recognition. Consequently, to focus on processing important emotional information, the speech-visual weights are calculated by counting the number of the extreme cases in the case (2) and case (3). Formally, the speech-visual weights and are defined as Based on and , this paper also proposes a simple and interpretable weighted fusion to focus on processing important emotional information, which is defined as where represents the fused features. and represents the speech-visual feature extraction networks by taking and as inputs, respectively. The selection of and are detailed in “Datasets and experimental setup”. Note that the speech-visual weights and in Eq.(10) are explicitly calculated by the number of the extreme cases in the case (2) and case (3). When the number of the extreme cases in the case (2) and case (3) increases, and become larger. This is more beneficial for speech-visual emotion recognition. Consequently, compared to implicitly learning weight fusion methods by training additional neural network layers on a large amount of data, the proposed weighted fusion method is explicitly derived from the number of the extreme cases, with weights adapting to the number of the extreme cases, thereby enhancing interpretability. Meanwhile, the first-stage SVKF alignment is summarized in Algorithm 2. First-Stage SVKF Alignment Experimental results and analysis Datasets and experimental setup The RML40, eNTERFACE0541, MEAD42 and BAUM-1s43 datasets are selected to evaluate the performance of the proposed method, in which the six basic discrete emotions are identified, including anger, disgust, fear, sadness, surprise and happy. The number of samples for each emotion on these four datasets is summarized in Table 1. Each emotional video in the four datasets is divided into multiple segments containing speech and visual modalities, where the label of the emotional video is viewed as the label of the multiple segments12. These emotional video segments with speech-visual information are fed into the proposed effective two-stage key frame extraction to obtain the second-stage SVKFs , and interpretable speech-visual weights , . This paper selects the wav2vec 2.0 model44 as the speech feature extraction network of . This is because the wav2vec 2.0 model is a transformer-based pre-trained model for self-supervised learning from numerous speech data, where it achieves better performance in speech emotion recognition. Meanwhile, this paper also selects the pre-trained spatio-temporal transformer (STT)45 as the visual feature extraction network of , since it is able to extract the spatio-temporal features of visual modality by incorporating both spatial and temporal attention. Meanwhile, a fully connected layer with 1024 neurons is added before the classification layers of the wav2vec 2.0 model and the pre-trained STT to guarantee speech-visual features with fixed dimensions46, i.e., , respectively. The speech-visual features and are fused by the proposed simple and interpretable weighted fusion, i.e., Eq.(eq16), where the fused features are fed into a three-layer MLP with the Softmax function to achieve speech-visual emotion classification. The PyTorch with the Adam optimizer is applied, where the learning rate and batch size are set to 0.001 and 64, respectively. Table 1The number of samples for each emotion.RMLeNTERFACE05MEADBAUM-1sAnger1202002874959Disgust1201892935786Fear1201872910538Sadness12019529349139Surprise1201902960943Happy12020529428179 The number of samples for each emotion. Comparison with state-of-the-art methods Tables 2, 3, 4 and 5 show the comparison results with other state-of-the-art algorithms, where ‘original frame extraction’ indicates that these state-of-the-art algorithms use the original paper method to acquire SVKFs, and ‘our frame extraction’ indicates that the proposed effective two-stage key frame extraction is used to acquire the second-stage SVKFs. More specifically, the method 1 extracts SVKFs in time-sequential order with equal probability, which leads to issues such as information redundancy and losing important information. The method 2 separately extracts SKFs and VKFs based on some key frame extraction strategies, but does not account for the alignment between SKFs and VKFs. The method 3 extracts SKFs and uses their corresponding visual information as VKFs, which does not fully account for the extreme cases of speech and visual modalities discussed in this paper. By comparing the accuracy of these comparison algorithms in the ‘original frame extraction’ and ‘our frame extraction’ on RML, eNTERFACE05, MEAD and BAUM-1s datasets, it can be seen that the proposed effective two-stage key frame extraction can be flexibly applied to other algorithms, in which the accuracy is also improved. This also demonstrates that the proposed effective two-stage key frame extraction has better generalizability. Table 2Comparative experimental results on RML dataset.Refs.Original frame extractionOur frame extractionZhang et al.1280.36% (Method 1)84.65%Ma et al.1882.38 % (Method 3)85.56%Cornejo et al.4782.50% (Method 1)85.82%Wozniak et al.4882.47% (Method 1)85.91%Kansizoglou et al.4982.97% (Method 1)86.58%Alasiry et al.2488.84% (Method 2)91.67%Chen et al.2096.79% (Method 1)98.83%Hina et al.5097.86% (Method 1)99.21%Chen et al.2198.16% (Method 1)99.47%Wu et al.2398.35% (Method 1)99.53% Comparative experimental results on RML dataset. Table 3Comparative experimental results on eNTERFACE05 dataset.Refs.Original frame extractionOur frame extractionWozniak et al.4872.27% (Method 1)78.43%Cornejo et al.4783.33% (Method 1)87.74%Alasiry et al.2483.98% (Method 2)88.24%Zhang et al.1285.97% (Method 1)89.39%Ma et al.1885.69% (Method 3)88.71%Chen et al.1987.20% (Method 1)89.93%Tang et al.5188.11% (Method 1)91.15%Nie et al.5297.07% (Method 1)99.11%Chen et al.2098.92% (Method 1)99.51%Chen et al.2299.63% (Method 1)99.89%Wu et al.2399.75% (Method 1)99.96% Comparative experimental results on eNTERFACE05 dataset. Table 4Comparative experimental results on MEAD dataset.Refs.Original frame extractionOur frame extractionWozniak et al.4851.29% (Method 1)58.69%Zhang et al.1258.96% (Method 1)63.21%Ma et al.1862.73% (Method 3)65.63%Pan et al.5363.15% (Method 2)66.22%Hina et al.5064.26% (Method 1)67.81%Chen et al.2073.24% (Method 1)77.15%Chen et al.2177.43% (Method 1)79.23%Bai et al.1477.51% (Method 1)79.34%Chen et al.2278.26% (Method 1)80.18%Wu et al.2378.31% (Method 1)80.22% Comparative experimental results on MEAD dataset. Table 5Comparative experimental results on BAUM-1s dataset.Refs.Original frame extractionOur frame extractionKansizoglou et al.4956.01% (Method 1)62.17%Ma et al.1859.17% (Method 3)64.98%Cornejo et al.4760.49% (Method 1)65.12%Pan et al.5360.49% (Method 2)65.12%Hina et al.5061.77% (Method 1)66.94%Chen et al.2071.26% (Method 1)75.36%Chen et al.2175.15% (Method 1)76.98%Bai et al.1476.29% (Method 1)78.13%Chen et al.2277.35% (Method 1)78.96%Wu et al.2378.59% (Method 1)79.21% Comparative experimental results on BAUM-1s dataset. Ablation experiments Table 6 shows the results of the proposed effective two-stage key frame extraction for different cases on the four datasets. The accuracy of the case (2) + case (3) is higher than those of the case (2) and the case (3), which indicates that bidirectional key frame supplementation outperforms unidirectional key frame supplementation. Further comparing the accuracy between all frames and the cases (1)-(3), the proposed effective two-stage key frame extraction improves the performance of speech-visual emotion recognition. This is because the proposed effective two-stage key frame extraction obtains the second-stage SVKFs to effectively reduce the redundancy of speech-visual key frames and save important emotional information, which ensures that the speech-visual frames inputted to the speech pre-trained wav2vec 2.0 model and the visual pr-trained STT are high-quality data. Table 6The results of the proposed effective two-stage key frame extraction for different cases.DatasetsAll framesCase (1)Case (2)Case (3)Case (2)+Case (3)RML94.38%99.56%99.81%99.76%99.86%eNTERFACE0596.62%99.52%99.84%99.79%99.90%MEAD72.69%79.25%81.23%81.85%82.74%BAUM-1s70.52%78.51%80.23%80.38%80.82% The results of the proposed effective two-stage key frame extraction for different cases. Table 7 shows the results of different feature fusion methods on the four datasets. NSVLF33, MSAFF34 and the proposed weighted fusion improve the accuracy of speech-vision emotion recognition, compared to the concatenation operator31,32. Although the accuracy of the proposed weighted fusion is slightly lower than that of MSAFF34, the weights of the proposed weighted fusion are explicitly derived from the number of the extreme cases in the case (2) and case (3), rather than being learned implicitly by training additional neural network layers on a large amount of data, which enhances interpretability. In the future, we will also integrate MSAFF34 into the proposed weighted fusion to simultaneously improve both the accuracy and interpretability of speech-visual emotion recognition. Table 7The results of different feature fusion methods.DatasetsConcatenation operator31,32NSVLF33MSAFF34Proposed weighted fusionRML97.38%98.44%99.68%99.65%eNTERFACE0598.12%98.8399.79%99.74%MEAD77.15%78.6380.41%80.34%BAUM-1s76.61%78.2179.41%79.36% The results of different feature fusion methods. Figures 3, 4, 5 and 6 show the confusion matrix of the proposed effective two-stage key frame extraction on these four datasets, respectively. As can be seen from the Figs. 3, 4, 5 and 6, emotions with higher intensity seem to be easier to identify. In particular, the accuracy of happy is 99.88% on the RML dataset, 99.89% on the eNTERFACE05 dataset, 81.15% on the MEAD dataset, 81.24% on the BAUM-1s datasets, and the accuracy of anger is 99.93% on the RML dataset, 99.94% on the eNTERFACE05 dataset, 83.56% on the MEAD dataset, 81.86% on the BAUM-1s datasets, which are higher than other emotion categories. In addition, emotions with similar intensity are easily misclassified, e.g., the result of misclassification of happy into anger is 0.12% on the RML dataset, 0.11% on the eNTERFACE05 dataset, 9.23% on the MEAD dataset, 12.62% on the BAUM-1s datasets, which are much higher than happy misclassification into other categories. The result of misclassification of fear into disgust is 0.26% on the RML dataset, 1.01% on the eNTERFACE05 dataset, 7.72% on the MEAD dataset, 9.6% on the BAUM-1s datasets, which are much higher than fear misclassification into other categories. Confusion matrix on RML dataset. Confusion matrix on eNTERFACE05 dataset. Confusion matrix on MEAD dataset. Confusion matrix on BAUM-1s dataset. Figure 7 shows the statistical results of the proposed effective two-stage key frame extraction method across 100 experiments with randomly shuffled data on four datasets. The fluctuation of the accuracy for the proposed effective two-stage key frame extraction method across the four datasets is less than 1%. This is because the proposed effective two-stage key frame extraction method reduces information redundancy in SVKFs through key frame extraction, ensuring that SVKFs fed into the speech and visual feature extraction networks maintain high quality even when the data is randomly shuffled, thereby reducing fluctuations in the accuracy. The statistical results of the proposed effective two-stage key frame extraction method. Conclusion This paper proposes the effective two-stage key frame extraction method for speech-visual emotion recognition. It obtains the first-stage SKFs to reduce the redundancy of speech key frames, and also extracts the first-stage VKFs to reduce the redundancy of visual key frames. By leveraging the complementarity characteristics between speech and visual modalities, the second-stage SVKFs are obtained by aligning the first-stage SKFs and VKFs, which effectively saves important emotional information. In addition, to focus on processing important emotional information, the feature layer fusion adopts the simple and interpretable weighted fusion, which also effectively improves the performance of speech-visual emotion recognition. Appendix: An example of the proposed effective two-stage SVKF extraction method Assuming there are 5 frames speech-visual information and , if the frames 2 and 3 belong to the first-stage SKFs, as well as the frames 2 and 4 belong to the first-stage VKFs, i.e., , , , , then three cases will exist based on the alignment of each frame according to the time steps (i.e., the position parameters in and ). More specifically, and correspond to the case (1) in Fig. 2, which are used as the second-stage SVKFs. correspond to the case (2) in Fig. 2, if the IE of is larger than or equal to the average IE of , the complemented speech frame and are used as the second-stage SVKFs, otherwise is deleted. correspond to the case (3) in Fig. 2, if the energy value of is larger than or equal to the average energy value of , and the complemented visual frame are used as the second-stage SVKFs, otherwise is deleted. Accordingly, for 5 frames speech-visual information and , the second-stage SVKFs are one of the following four. If the IE of is larger than or equal to the average IE of , and the energy value of is larger than or equal to the average energy value of , the second-stage SVKFs are and , and the complemented visual frame in the case (3), as well as the complemented speech frame in the case (2) and . If the IE of is larger than or equal to the average IE of , and the energy value of is larger than or equal to the average energy value of , the second-stage SVKFs are and , and the complemented visual frame in the case (3), as well as the complemented speech frame in the case (2) and . If the IE of is larger than or equal to the average IE of , and the energy value of is smaller than the average energy value of , the second-stage SVKFs are and , as well as the complemented speech frame in the case (2) and . If the IE of is larger than or equal to the average IE of , and the energy value of is smaller than the average energy value of , the second-stage SVKFs are and , as well as the complemented speech frame in the case (2) and . If the IE of is smaller than the average IE of , and the energy value of is larger than or equal to the average energy value of , the second-stage SVKFs are and , as well as and the complemented visual frame in the case (3). If the IE of is smaller than the average IE of , and the energy value of is larger than or equal to the average energy value of , the second-stage SVKFs are and , as well as and the complemented visual frame in the case (3). If the IE of is smaller than the average IE of , and the energy value of is smaller than the average energy value of , the second-stage SVKFs are and . If the IE of is smaller than the average IE of , and the energy value of is smaller than the average energy value of , the second-stage SVKFs are and . Author contributions Yuanhui Liang: Conceptualization, Methodology, Software, Investigation, Formal analysis, Writing - original draft. Xiaojing Li: Methodology, Software, Investigation, Formal analysis. Lei Wang: Visualization, Investigation. Funding This work is funded by “Henan Province’s New Key Discipline - Machinery” Project No.2023414. Data availability The datasets used and/or analysed during the current study available from the corresponding author on reasonable request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.