Deep Learning for Speech Emotion Recognition: A CNN Approach Utilizing Mel Spectrograms
Niketa Penumajji · 2025
Read Deep Learning for Speech Emotion Recognition: A CNN Approach Utilizing Mel Spectrograms by Niketa Penumajji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19677.txt.
Introduction
Niketa Penumajji teaches a neural network to hear feelings in speech, in a 2025 paper. Older statistical methods for emotion recognition proved too weak for real use. The approach here turns each audio clip into a Mel spectrogram, a picture of how pitch energy spreads over time. A convolutional network then learns the patterns that mark anger, joy, sadness, and the like on its own.
The trained model sits inside a friendly graphical interface that predicts emotions in real time. Penumajji weighs classroom uses, where a system that senses frustration or engagement could help teachers respond. The study checks the method's feasibility end to end, from sound file to live tool.
Catalog record
| CREATOR | Niketa Penumajji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19677
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deep Learning for Speech Emotion Recognition: A CNN Approach Utilizing Mel Spectrograms Niketa Penumajji Abstract This paper explores the application of Convolutional Neural Networks (CNNs) for classifying emotions in speech through Mel Spectrogram representations of audio files. Traditional methods such as Gaussian Mixture Models and Hidden Markov Models have proven insufficient for practical deployment, prompting a shift towards deep learning techniques. By transforming audio data into a visual format, the CNN model autonomously learns to identify intricate patterns, enhancing classification accuracy. The developed model is integrated into a user-friendly graphical interface, facilitating real-time predictions and potential applications in educational environments. The study aims to advance the understanding of deep learning in speech emotion recognition, assess the model’s feasibility, and contribute to the integration of technology in learning contexts. I Introduction Traditional approaches to the incorporation of technology in learning environments have tended to focus more on interactivity as opposed to analysis, which has resulted in a lack of progress in the areas of performance review, analysis, and overall utility (Cho et al. 2020). This is due in part to the complexity of such environments, with many subjective metrics and mediums that, at first glance, appear to be difficult to study and quantify. In this project, speech emotion has been identified and selected as a suitable metric for analysis. Speech emotion is a language agnostic metric that transcends many of the barriers posed by the more obvious metrics such as spoken / body language or facial expressions. Despite the many nuances of speech, such as tonality or pitch, there are clear patterns that can be derived from studying how emotions are conveyed (Angrick et al. 2019). A spectrogram is a visual representation of the spectrum of frequencies (in this case, the frequencies are converted to the mel scale) across a signal as a function of time. Consider the following mel spectrograms produced from two different male actors saying a short sentence in a blatantly angry and then sad manner. Fig. 1: Spectrogram comparison Even from a macro perspective, it is possible to make out some visible similarities. Take for example the large circular shapes that can be made out from the two angry clips, or the distribution of the yellow (corresponding to pitch across the mel scale) lines in the sad clips. The spectrogram is represented by a matrix of values, which is where the vast, more subtle patterns can be identified and observed - making it an ideal candidate for a CNN approach. Prior to the increased popularity of machine learning, conventional methods for analysing and processing speech involved the usage of Gaussian Mixture Models (GMM) and Hidden Markov Models (HMM), both of which produced results that were simply too inaccurate to ever put into production (Sainath et al., 2013; Venkataramanan & Rajamohan, 2019). The accessibility of machine learning techniques has lead to novel approaches such as the application of Recurrent Neural Networks (RNN) and CNNs, resulting in improved accuracy and introducing completely different approaches to the problem. In this paper, inspired by the notable success and improvements seen from the application of CNNs in particular, a relatively accurate classification model has been trained by applying CNNs to Mel Spectrogram representations of audio files. The Mel Spectrogram effectively allows me to transform the classification problem from the audio domain into the visual domain, where CNNs have demonstrated to excel. As previously shown in Figure 1, there are many subtle patterns to be found in the data, but it is a difficult task to calculate and label these ourselves manually. Hence, a deep learning approach, where the model learns to recognize and extract features itself, becomes a clear suitable choice. The final trained model has then been deployed into a general purpose graphical tool that can perform predictions across pre-recorded audio files or through recording on the fly. The graphical tool has been purposefully designed with simplicity in mind and with the intention of being easy to use and incorporate into learning environments. II Research Objectives • To research, experiment, and develop an understanding with deep learning using industry standard frameworks. • To develop a plausible speech emotion classifier using state of the art techniques. • To deploy the speech emotion classification model into an accessible GUI application. • To assess the feasibility of such a tool for use in learning environments. III Literature Review III-A Automatic Speech Emotion Recognition Using Machine Learning Automatic Speech Emotion Recognition Using Machine Learning explores SER systems, using Mel-frequency cepstrum coefficients and modulation spectral features with feature selection while comparing machine learning approaches. It evaluates an RNN against multivariate linear regression and support vector machines, chosen for their popularity, but omits CNNs, which could have been relevant. Despite this, the paper is well-structured, clear, and accessible to readers with limited prior knowledge. It highlights the advantages of SER in detecting emotional states, especially in learning environments, and details the authors’ system, including dataset constraints and algorithm selection. The study concludes that SER performs best with limited data, while RNNs are more effective with larger datasets—insights that are valuable for assessing the effectiveness of my SER system. III-B A CNN-Assisted Enhanced Audio Signal Processing for Speech Emotion Recognition Due to the notable omission of a CNN approach in the first reviewed paper, this paper has primarily been selected to gain practical insight into such an approach. This paper explores the CNN approach to SER, focusing on extracting hidden information (Kwon, 2020) and taking a more human-computer interaction (HCI) perspective than the previous paper’s theoretical approach. While it clearly explains its methodology, it assumes prior knowledge of machine learning concepts. The proposed model processes 2D speech spectrograms through convolutional layers and a softmax classifier, requiring less pre-processing than the first paper’s methods. However, it lacks technical depth, with minimal equations and limited training. Despite this, it offers valuable insights, making it a useful comparison to the previous study. III-C Emotion Recognition from Speech Emotion Recognition from Speech has been instrumental in shaping the classifier architecture for this project. It compares various industry-standard SER methods, including HMMs, MFCCs, and CNNs, finding that a four-layer model with batch normalization and max pooling achieves the highest accuracy. It also covers data processing techniques like Wiener filtering for noise reduction. Overall, the paper offers valuable architectural insights and comparisons that have greatly influenced this project’s development. III-D Factors Affecting Technology Integration in the Classroom This paper outlines five key barriers to technology integration in classrooms: poor infrastructure, inadequate technology, lack of professional development, low self-efficacy, and teacher perceptions. The most relevant to this project are inadequate technology and teacher perceptions. Harrel and Bynum (2018) highlight that teachers assess both the effort required to learn new technology and its practicality before adopting it. Since this project aims to propose a viable application for learning environments, gaining educator buy-in will be essential. III-E Modeling learners’ cognitive and affective states to scaffold SRL in open-ended learning environments Whilst this paper is not directly related to the project, it offers valuable insights into the relationship between cognitive and affective states, which will aid in the analysis phase. Specifically, the paper highlights a correlation between boredom and delight and academic performance, noting that boredom significantly reduces