EmoAra: Emotion-Preserving English Speech Transcription and Cross-Lingual Translation with Arabic Text-to-Speech Besher Hassan, Ibrahim Alsarraj, Musaab Hasan, Yousef Melhim, Shahem Fadi, Shahem Sultan, Email: Besher.Hassan@mbzuai.ac.ae, Ibrahim.Alsarraj@mbzuai.ac.ae Abstract This project addresses challenges in cross-linguistic communication, particularly in banking customer service, where understanding emotional context is essential. By addressing language barriers, fostering empathetic communication, and enhancing client satisfaction, the system demonstrates practical applications in banking customer service. It integrates Speech Emotion Recognition (SER) , Automatic Speech Recognition (ASR) , Machine Translation (MT) , and Text-to-Speech (TTS) to enable emotion-preserving, multilingual communication. Using a CNN for emotion detection, Whisper for English transcription, MarianMT for English-to-Arabic translation, and MMS-TTS-Ara for Arabic speech synthesis, the system processes spoken English, detects emotions, transcribes and translates the text, and delivers a synthesized Arabic response while retaining emotional nuance. Key findings include an F1-score of 94% for emotion detection, a BLEU score of 56 , a BERTScore F1 of 88% for translation quality, and 81% in human evaluation for translation accuracy, fluency, and domain-specific terminology. For further details, check the GitHub Repository . 1 Introduction 1.1 Project Overview In an increasingly globalized world, businesses face the challenge of communicating effectively across language and cultural barriers. The goal of this project is to build an emotion recognition system for applications in customer service specifically within the banking sector. By extracting the emotion from a customer’s spoken interaction, and then transcribe the English audio, translate the text into Arabic, and use the text-to-speech (TTS) system for oral presentation. Combine Emergent Speech Emotion Recognition (emergent SER), Automatic Speech Recognition (ASR), Machine Translating (MT) with text-to-speech (TTS) and create multilingual communication, retaining emotional intactness. The project will be language-agnostic and emotion-oriented. Emotion-oriented systems are systems designed to recognize and process content to understand and preserve the emotional context in communication. In customer service contexts, it is common to look at the emotional tone in a customer’s voice which helps in gauge what the customer needs and feels in particular. The importance gets magnified in case of banking since emotions help in Assessing the degree of urgency, satisfaction, or dissatisfaction. Unfortunately, language barriers and lack of live emotional responses act as a hindrance in enabling the bank representatives from properly addressing the issues on time. Our project utilizes emotion and language understanding to ease these barriers and give the industry the ability to respond emotionally to emotionally charged speech in any language. 1.2 Primary Contributions the primary contributions of this project can be summarized as: • Emotion in Cross-Linguistic Interactions: In contrast to many existing systems which ignore emotional tone in translations, translation done in this project has an emotional concern as to the fact that emotions in the original language will definitely be present in the target language as well, making clients feel more natural and more empathy in the service provision. • Practical Application in Banking Customer Service: A typical application of our system falls within the banking and finance industry where communication across different languages for effective service delivery is possible as well as the need to understand customers emotions. 1.3 Related Work Doctors Livingstone and Russo (2018) provided the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), which has now become a widely used benchmark for assessment of SER systems ( livingstone2018ryerson ) . Employing this dataset, researchers have used CNNs and RNNs for emotion recognition for a wider variety of emotions. Huang et al. (2021) showed how models for emotion detection can be fine-tuned through the transfer learning process towards achieving better accuracy by retraining existing models ( huang2021speech ) . Hannun et al. (2014) introduced the DeepSpeech system which leveraged recurrent neural networks trained on massive datasets of speech and achieved impressive results ( hannun2014deepspeech ) . Lately, Baevski et al. (2020) presented wav2vec 2.0 which enables the self-training of networks which produce better speech representations than what has been previously achieved using end-to-end architectures ( baevski2020wav2vec ) . Machine Translation (MT) has made one of the most rapid advancements as well with special emphasis on specific low-overhead language models. Junczys-Dowmunt et al. (2018) introduced MarianMT system which is a quick, powerful, and elegant neural machine translation system and achieved the best results in neural translation for several language pairs ( junczys-dowmunt2018marian ) . Text-to-Speech (TTS) systems have developed over the years with the help of technologies such as Google Cloud’s TTS service which produces nearly natural speech. Several languages including Arabic are supported and systems like these have been used in a number of practical cases. But still, the majority of existing TTS systems are designed in such a way that their functions are separated from systems of emotion recognition and machine translation. This, in turn, renders these systems less effective for dynamic and emotionally complex applications ( google2021tts ) . Building on these ideas and undertaking the relevant research work, this project seeks integration of ASR, MT, SER, and TTS methods into a single, coherent system. In particular, we want to create a powerful end-to-end multilingual system that is capable of preserving the emotionality of the content during the language conversion pipeline through the application of transfer-learning methods and pretrained models. 2 Problem Statement In the present interconnected world, it is crucial that people who offer a service and the customers know how to effectively communicate across different languages particularly in fields like banking where the customer’s emotion can be valuable in determining the service offered. Unfortunately, most automated customer service systems are unable to detect and interpret emotions embedded in speech in the first and different languages. Such a impediment may contribute to miscommunication, less satisfaction of the customer and failure to engage in responsive communication. Our objective is to address this gap by developing a system that not only transcribes and translates spoken language but also retains the emotional context within customer interactions. By implementing a multilingual, emotion-driven speech recognition and translation pipeline, our solution will empower customer service teams to respond more empathetically and accurately to customers’ needs, regardless of language differences. 3 Proposed Method 3.1 Exploratory Data Analysis 3.1.1 Translation Data Preprocessing To prepare the translation datasets for fine-tuning the Marian MT model, preprocessing steps were applied to make the data compatible with the model: • Text Normalization : Lowering case, deleting numbers and punctuation, and cutting excessive text space. • Language-Specific Filtering : To warrant consistency of the data submitted, only the Arabic script was kept on the Arabic data and English alphabetic letters only in English data. • Tokenization and Padding : Sentences were tokenized using MarianMT tokenizer, then truncating sequences to a maximum length of 128 tokens to achieve a balance between memory efficient and text coverage. These preprocessing steps prepared the datasets for fine-tuning by standardizing the content, maintaining linguistic consistency, and reducing errors during training. 3.1.2 Audio Files There are 1440 audio files in .wav format. The audio files tagged with ‘angry‘ and ‘calm‘ have distinctly different frequency and amplitude characteristics as seen in Figure 1 . The left spectrogram illustrates the log-frequency power spectrum of an ”Angry” speech sample, characterized by higher intensity and wider frequency variations. The right spectrogram represents the ”Calm” emotion, showing lower intensity and smoother frequency transitions. These differences highlight the distinct acoustic features of emotional speech, particularly in terms of energy distribution across frequencies and time. Figure 1: Log-Frequency Power Spectrum of Different Emotions 3.2 Method We built an end-to-end system that consists of four components as shown in Figure 2 , CNN model for extracting the emotion. ( LSTM and ResNet50 models were implemented and tested instead of CNN, more details can be found in A ). Followed by Whisper openai_whisper pre-trained transformer model to recognize the speech and convert it to text. Then the English sentences are translated into Arabic sentences using MarianMT huggingface_marian pre-trained fine-tuned transformer model. Finally, the text will be converted to speech using a MMS-TTS-Ara facebook_mms_tts_ara pre-trained model. Their respective roles in the processing pipeline: Figure 2: System Architecture 3.2.1 CNN Model for Speech Emotion Recognition: This component employs an artificial neural network model that is created in TensorFlow/Keras to recognize emotion in audio data. The following was implemented: Data Preparation: In order have better generalization, numerical features from (emotion-features.csv) was created and data augmentation techniques were applied to the audio files to generate multiple variations of the same audio file to simulate different acoustic conditions. Below are the applied techniques: • Adding Noise: Gaussian noise is added to the audio signal. y_{\text{noisy}}=y+\text{noise\_factor}\cdot\mathcal{N}(0,1) (1) where y is the original signal and \mathcal{N}(0,1) is a standard normal distribution. • Pitch Shifting: Adjusts the pitch of the signal by shifting it up or down in frequency. • Time Stretching: Speeds up or slows down the audio signal. • Time Shifting: Shifts the signal by a percentage of its length. Following the data augmentation, feature extraction was used to convert the audio waveforms into numerical representations, which is a dimensionality reduction step for keeping only the relevant information of the data. Several feature extraction approaches were used: • Zero-Crossing Rate (ZCR) By measuring the rate at which the signal crosses the zero amplitude line, capturing the noisiness of the signal. \text{ZCR}=\frac{1}{T}\sum_{t=1}^{T}1\left[y_{t}\cdot y_{t-1}<0\right] (2) where T is the number of samples, and 1 is an indicator function. • Root Mean Square Energy (RMSE) Measures the energy of the audio signal. \text{RMSE}=\sqrt{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}} (3) • Mel-Frequency Cepstral Coefficients (MFCCs) Captures the spectral properties of the signal and represents its timbre. \text{MFCC}[k]=\sum_{n=0}^{N-1}\log(X[n])\cdot\cos\left(\frac{N\pi k}{N}(n-0.5)\right) (4) where X[n] is the Mel-scaled power spectrum. After the feature extraction, the data will be ready to be fed into the CNN model. The construction of CNN is as the following: One-Hot Encoding: Emotion labels (Y) were categorical and were one-hot encoded using OneHotEncoder. This converts the labels into binary vectors which are suitable for multi-class classification. Feature Scaling: All features were normalized using the (StandardScaler), it scales the data to have zero mean and unit variance to ensure the numerical stability during optimization. Reshaping for CNN Input: The features got reshaped to (samples,features,1), making them compatible with the 1D convolutional layers. The CNN model was designed with the following layers: Input Layer: Accepts input data of shape (features,1) which represents the scaled features. Convolutional Layers: The audio signals undergo three 1D convolutional layers (with filters) that extract features with ReLU activations to non-linearities that are used to address vanishing gradients. Batch Normalization: Normalizes the outputs of each layer to accelerate training. Max Pooling and Dropout Layers: Each of the convolution layers has max pooling as a subsequent layer in a bid to reduce the feature dimensionalities, and dropout for convectional layers and for dense layers to prevent overfitting. Fully Connected Dense Layers: After the convolutional output has been flattened, a dense layer of 256 neurons has been used with ReLU activation and the softmax output is done in multi-class after classifying the images. The model is compiled using the Adam optimizer, which is known for its efficiency in training deep neural networks. Categorical cross-entropy is used as the loss function, as the problem is a multi-class classification task. Overall, CNNs are effective for capturing patterns in sequential data. The use of 1D convolution layers matches the structure of the audio features. 3.2.2 Whisper by OpenAI Whisper is an advanced automatic speech recognition (ASR) system developed by OpenAI openai_whisper . Its architecture is based on a transformer-based model. Key components: Encoder-Decoder Architecture: Whisper adopts an encoder-decoder approach where in this case the encoder is responsible for the raw audio while the decoder generates the transcription from the encoded features. Using this sequence-to-sequence framework, the decoder takes in the speech signal and produces text tokens. For our project, we are using Whisper Base openai_whisper_github . Different Whisper variants offering speed and accuracy trade-offs. In general, Whisper is very robust against variations like accents, noise, and background disturbances which makes it a reliable ASR model. 3.2.3 English To Arabic Translation In this section, the focus is to enhance English-to-Arabic translations in the banking field through the use of a fine-tuned Marian MT. First, attempts to provide translation through multilingual models were made. Specific steps that included evaluation of pre-training, preparation of training dataset, fine-tuning and, evaluation metrics incorporating various metrics were utilized. Domain-specific data and advanced transformer-based architecture has been use. Dataset Preparation The dataset included a custom created banking translation dataset and an English to Arabic general translation dataset. These were divided into training, validation, and testing subsets with the proportions of 80%, 10% and 10% respectively. Preliminary Model and Challenges The objective of this work was to develop a high-performing translation model. Due to the limited dataset size, training a custom-built Transformer model posed significant challenges. Initial efforts involved training a Transformer model on 24,000 English–Arabic sentence pairs. This approach provided basic translation capabilities but was constrained by the dataset’s size, which was insufficient for effective generalization—particularly for complex language pairs like English and Arabic. Building a high-performing Transformer model typically requires millions of sentence pairs to achieve the desired results. To address these constraints, we fine-tuned a pre-trained model, MarianMT, to optimize translation performance on the available small-scale dataset. Adoption of Marian MT Marian MT, a pre-trained translation model by Helsinki-NLP helsinki-nlp , is a transfomer-sequence-model highly efficient based. Another way to consider it is that it consists of an encoder, a decoder and multi-head self attention mechanism and layer normalization and position-wise feed forward networks. The source language text about contextual and syntactic structures is captured by the encoder as a series of continuous representations. Then, the encoder transforms the source text into several continuous representations. The decoder then generates a target text by attending to the representations and modeling the dependencies in a target sequence. This has emphasized the importance of pre-trained models and extra data this made MariaMT as a suitable choice. Hyperparameter Experimentation In order to undertake training for the Marian MT model optimization, experiments with some key hyperparameters were carried out, such as learning rate, batch size, number of epochs, beam search width, and gradient accumulation steps. and several configurations were tested to find the optimal hyperparameters values that gives the best translation quality. Mathematical Formulation The model minimizes the cross-entropy loss \mathcal{L} between the predicted tokens \hat{y} and the reference tokens y , defined as: \mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}\log P(\hat{y}_{i}|y_{