Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization

Hsuan-Yu Wang Department of English National Taiwan Normal University Taipei, Taiwan [email protected] Pei-Ying Lee Department of Computer Science and Information Engineering National Taiwan Normal University Taipei, Taiwan [email protected] Berlin Chen Department of Computer Science and Inf · 2025

Read Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization by Hsuan-Yu Wang Department of English National Taiwan Normal University Taipei, Taiwan [email protected] Pei-Ying Lee Department of Computer Science and Information Engineering National Taiwan Normal University Taipei, Taiwan [email protected] Berlin Chen Department of Computer Science and Inf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19356.txt.

Introduction

Hsuan-Yu Wang, Pei-Ying Lee, and Berlin Chen improve speech emotion recognition by aligning transcripts with speaker labels. Their 2025 paper works on conversations, where a transcript from speech recognition and a diarization output often fall out of step. That mismatch weakens systems that judge emotion from both text and sound.

Their pipeline uses pre-trained recognition and diarization models, then synchronizes their timestamps to make cleanly labeled speaker segments. Text features from a RoBERTa model are fused with audio features through cross-attention with a gate. On the IEMOCAP benchmark, precise alignment beat baselines that skip synchronization. The authors argue that temporal alignment is not a detail but a core part of multimodal emotion analysis.

Catalog record

CREATORHsuan-Yu Wang Department of English National Taiwan Normal University Taipei, Taiwan [email protected] Pei-Ying Lee Department of Computer Science and Information Engineering National Taiwan Normal University Taipei, Taiwan [email protected] Berlin Chen Department of Computer Science and Inf
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-19356

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization Hsuan-Yu Wang Department of English National Taiwan Normal University Taipei, Taiwan [email protected] Pei-Ying Lee Department of Computer Science and Information Engineering National Taiwan Normal University Taipei, Taiwan [email protected] Berlin Chen Department of Computer Science and Information Engineering National Taiwan Normal University Taipei, Taiwan [email protected] Abstract In this paper, we investigate the impact of incorporating timestamp-based alignment between Automatic Speech Recognition (ASR) transcripts and Speaker Diarization (SD) outputs on Speech Emotion Recognition (SER) accuracy. Misalignment between these two modalities often reduces the reliability of multimodal emotion recognition systems, particularly in conversational contexts. To address this issue, we introduce an alignment pipeline utilizing pre-trained ASR and speaker diarization models, systematically synchronizing timestamps to generate accurately labeled speaker segments. Our multimodal approach combines textual embeddings extracted via RoBERTa with audio embeddings from Wav2Vec, leveraging cross-attention fusion enhanced by a gating mechanism. Experimental evaluations on the IEMOCAP benchmark dataset demonstrate that precise timestamp alignment improves SER accuracy, outperforming baseline methods that lack synchronization. The results highlight the critical importance of temporal alignment, demonstrating its effectiveness in enhancing overall emotion recognition accuracy and providing a foundation for robust multimodal emotion analysis. I Introduction Speech Emotion Recognition (SER) has gained substantial research attention, particularly for its applications in human-computer interaction. While significant advancements have been made, real-world conversational scenarios pose unique challenges. Traditional SER systems often rely on manually segmented utterances, which are impractical to obtain at scale. Consequently, recent efforts have focused on integrating components like Speaker Diarization (SD) and Automatic Speech Recognition (ASR) to enable more autonomous emotion analysis from raw audio [ 1 ] . However, a fundamental limitation persists within these integrated approaches: the lack of precise temporal synchronization between ASR transcripts and speaker diarization outputs. This inherent misalignment can severely compromise the reliability of multimodal emotion recognition systems, especially in dynamic, turn-taking dialogues. To overcome this critical challenge, we introduce a timestamp alignment pipeline that systematically matches ASR transcripts with speaker diarization segments. This process ensures that each transcribed sentence is accurately matched to its speaker and temporal boundaries, yielding high-quality, speaker-attributed utterances. Using this aligned data, our multimodal SER system combines RoBERTa-based textual embeddings [ 2 ] with wav2vec 2.0-derived acoustic features [ 3 ] , wherein a cross-attention fusion mechanism, augmented by a gating component, integrates these modalities effectively [ 4 ] . Evaluations on the IEMOCAP dataset [ 5 ] demonstrate that precise alignment significantly improves SER accuracy, outperforming baseline systems lacking synchronization. These results emphasize the essential role of alignment in enhancing the robustness and reliability of multimodal emotion recognition. II Related Work A recent line of research has addressed the limitations of manual segmentation in Speech Emotion Recognition (SER) by integrating Automatic Speech Recognition (ASR), speaker diarization (SD), and Voice Activity Detection (VAD) into fully automatic pipelines. While such integration improves scalability, it introduces challenges—particularly due to temporal misalignment between outputs from these components. Whisper [ 6 ] , a self-supervised ASR model with fine-grained timestamping, has facilitated more precise speaker attribution and segment alignment. Building on this, [ 7 ] demonstrated that even minor misalignments between ASR transcripts and speaker turns can significantly degrade SER performance. These works underscore the importance of synchronization but fall short of providing flexible, modular approaches that split alignment from end-to-end learning. Meanwhile, a number of works have improved SER through architectural and learning-based innovations. [ 8 ] introduced a NAS-based framework to jointly optimize CNN and sequential modules, while [ 9 ] and [ 10 ] showed that pre-trained ASR features and compact representation learning respectively enhance emotion classification. [ 11 ] advanced modality fusion via a multi-loss framework, and [ 12 ] incorporated conversational context in dyadic settings. [ 13 ] emphasized the need to model temporal dynamics in emotion progression. These efforts contribute to downstream accuracy. However, these works often assume well-aligned inputs and overlook the critical preprocessing stage. Our approach addresses this upstream alignment challenge—an implicit dependency in many SER models. Fig. 1: Proposed Model Specific attention has also been paid to joint modeling of ASR and SD. [ 14 ] proposed an end-to-end speaker-attributed ASR model, while another approach [ 15 ] uses encoder-decoder attractors to jointly infer transcripts and speaker turns. [ 16 ] formally introduced the Speech Emotion Diarization (SED) task and proposed the EDER metric to assess time-sensitive accuracy. [ 17 ] demonstrated that syntactic alignment cues further improve speaker attribution in realistic meetings. While effective, these methods are often tightly coupled or application-specific. In contrast, our modular alignment pipeline can be flexibly integrated with pre-trained ASR and SD models to enable more robust and transferable multimodal SER. III Proposed Model This section sheds light on our proposed alignment-based multimodal SER framework, designed to robustly recognize speech emotions in conversational contexts. Our framework is structured around two core sections: a precise timestamp-based alignment pipeline for ASR and SD outputs, and a sophisticated cross-attention fusion mechanism for multimodal embeddings. The overall architecture is schematically depicted in Figure 1. III-A ASR, SD, and Alignment Approach Accurate temporal synchronization between spoken words and their corresponding speakers is essential for reliable multimodal emotion recognition in conversational contexts. Our system begins by processing raw input audio through two independent, state-of-the-art models: WhisperX for ASR [ 6 ] and Pyannote 3.1 for SD [ 18 ] . WhisperX generates detailed word-level transcripts with precise timestamps (e.g., [6.92s → 7.16s] ”Excuse”, [7.16s → 7.23s] ”me”), while Pyannote identifies speaker turns by producing timestamped speaker-labeled segments (e.g., [6.92s → 7.24s] Speaker_00). These two outputs serve as the foundation for our alignment procedure, which systematically links the transcribed text to the appropriate speaker segments. The entire process is visualized in Figure 2. Fig. 2: Timestamp Alignment Block To overcome the fragility of sentence-level text matching and mitigate data loss from ASR errors, we developed a robust, time- and speaker-aware alignment pipeline. This process, illustrated in Figure 2, transforms the word-level ASR and diarization outputs into coherent conversational turns. The procedure unfolds in three main stages: • Flatten Word Stream: The process begins by integrating the output from the upstream ASR and diarization stage. All speaker-attributed words, complete with their individual timestamps, are extracted from their initial segments and organized into a single, chronologically ordered stream. This step effectively dissolves the preliminary, and often fragmented, ASR segment boundaries to create a continuous flow of word-level data. • Group Words into Turns by Speaker & Pause: Next, this flat stream of words is grouped into semantically and contextually coherent turns. The grouping algorithm iterates through the words, merging consecutive words into a single turn as long as two conditions are met: (1) the speaker label remains consistent, and (2) the temporal pause between consecutive words does not exceed a predefined threshold (e.g., 1.5 seconds). A new turn starts whenever a speaker change or a significant pause is detected. This ensures that natural conversational breaks are respected while maintaining contextual continuity. • Construct Turn-level Segments: Finally, each grouped turn is finalized into a single, unified segment. The full text of the segment is constructed by joining the constituent words. Its overall timestamp is defined by the start time of the first

Read the complete text