Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset

Rui, Pu, Jiatian, Berrak, Carlos, Haizhou · 2025

Read Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset by Rui, Pu, Jiatian, Berrak, Carlos, Haizhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20341.txt.

Introduction

Rui Liu and co-authors tackled emotion in text-based speech editing in a 2025 paper. Such editing changes spoken words by editing text, with no re-recording. Past methods cared about content and sound quality but often let emotion drift or clash after a text change.

Their system, EmoCorrector, works after editing to fix emotion. It reads the edited text's emotional traits, retrieves speech samples with matching emotion, and synthesises speech that fits while keeping the speaker's voice. The team also built a benchmark data set with varied texts and emotions. Tests showed the system greatly improved the expression of the intended emotion.

Catalog record

CREATORRui, Pu, Jiatian, Berrak, Carlos, Haizhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20341

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset Rui, Pu, Jiatian, Berrak, Carlos, Haizhou Abstract Text-based speech editing (TSE) modifies speech using only text, eliminating re-recording. However, existing TSE methods, mainly focus on the content accuracy and acoustic consistency of synthetic speech segments, and often overlook the emotional shifts or inconsistency issues introduced by text changes. To address this issue, we propose EmoCorrector, a novel post-correction scheme for TSE. EmoCorrector leverages Retrieval-Augmented Generation (RAG) by extracting the edited text’s emotional features, retrieving speech samples with matching emotions, and synthesizing speech that aligns with the desired emotion while preserving the speaker’s identity and quality. To support the training and evaluation of emotional consistency modeling in TSE, we pioneer the benchmarking Emotion Correction Dataset for TSE (ECD-TSE). The prominent aspect of ECD-TSE is its inclusion of < < text, speech > > paired data featuring diverse text variations and a range of emotional expressions. Subjective and objective experiments and comprehensive analysis on ECD-TSE confirm that EmoCorrector significantly enhances the expression of intended emotion while addressing emotion inconsistency limitations in current TSE methods. Code and audio examples are available at https://github.com/AI-S2-Lab/EmoCorrector . Liu Gao Xi Sisman Busso Li HohhotChina Johns Hopkins UniversityUSA Carnegie Mellon UniversityUSA The Chinese University of Hong KongShenzhen, China National University of SingaporeSingapore keywords Text-based Speech Editing, Emotional Consistency, Retrieval-Augmented Generation (RAG), Emotion Post-Correction † † email: [email protected] 1 Introduction Text-based Speech Editing (TSE) modifies audio by editing its underlying text rather than the audio signal directly. With the rise of digital media, TSE has become essential for applications like social media content creation , game voiceovers , and film dubbing , as it corrects issues such as mispronunciations, omissions, or stuttering without requiring a full re-recording [ 1 ] . Recent advancements in text-to-speech (TTS) have led to the development of neural models for TSE [ 1 , 2 , 3 , 4 , 5 ] . For example, Despite these advancements, most TSE methods focus on content modification and acoustic quality, often neglecting emotional consistency [ 6 ] . As shown in Fig. 1 , although there is only one word difference between the edited text and the original text, the emotional expression at the sentence level is very different. Although the traditional TSE model successfully synthesizes the speech segment for the word “bad”, it does not express the emotion that the edited text should contain. At this point, the emotional state of the whole sentence needs to be further corrected to achieve emotional consistency. Figure 1 : Our approach lies in correcting the emotional mismatch or inconsistency issue of traditional TSE methods. A natural idea is to use Emotional Voice Conversion (EVC) [ 7 , 8 ] models to transform the emotion of an entire sentence into the same emotional state that the edited text contains without altering the speaker’s identity. However, directly using an EVC model presents several issues. 1) Before applying EVC, it is necessary to identify the emotional state embedded in the edited text, a process that may introduce errors. 2) Even if the identified emotion state is accurate, using discrete emotion labels as input for the EVC model may fail to capture rich and nuanced emotional information, thereby limiting the model’s performance. 3) The output of the EVC model is inherently uncontrollable, making it difficult to evaluate whether the converted results truly achieve optimal performance. Therefore, directly relying on an EVC model for post-correction does not fully meet our requirements. To address the above issues, we propose an end-to-end post-correction approach to rectify emotional inconsistencies for TSE, termed EmoCorrector. Our EmoCorrector is inspired by the concept of Retrieval-Augmented Generation (RAG) [ 9 ] . First, emotional embedding are extracted from the edited text, and speech samples with similar emotional embedding are retrieved from the cross-modal retrieval database. Next, the emotional embeddings of these retrieved speech samples are used as a reference, while the speaker features from the original edited speech are retained to preserve speaker identity. Finally, TTS synthesis is performed, enabling the modification of the original edited speech’s emotional information and achieving consistent emotional expression. For the feature construction of the cross-modal database, we adopt a contrastive learning [ 10 ] strategy focused on emotional features to perform language-audio contrastive learning, aligning the emotional features of text and speech within a unified space. Additionally, we design a speaker-emotion feature disentanglement strategy [ 11 ] based on adversarial learning [ 12 ] to ensure the independence of emotional and speaker feature representation. To support training and evaluation for emotional post-correction, we construct a new Emotion Correction Dataset for TSE (ECD-TSE). This dataset includes original text as well as edited text reflecting different emotional states. We utilized three advanced TTS models to perform emotional speech synthesis for all the text. This emotion-rich speech database, containing text editing information, provides a solid foundation for modeling emotional consistency. Extensive experiments on ECD-TSE demonstrate that EmoCorrector significantly enhances emotional alignment while maintaining natural audio quality. The main contributions of this paper can be summarized as follows: 1) We propose a comprehensive end-to-end emotion post-correction scheme for TSE, named EmoCorrector. 2) Our EmoCorrector draws on the techniques of RAG and designs cross-modal emotion contrastive learning and speaker-emotion disentanglement learning strategies. 3) We introduce the groundbreaking ECD-TSE dataset, which advances emotional consistency modeling and evaluation in the field of TSE. 2 Dataset: ECD-TSE For the emotionally consistent modeling of TSE, we need a database where sentences conveying similar lexical information with few modifications elicit different emotions. Unfortunately, popular databases such as the ESD dataset [ 13 ] , MSP-PODCAST [ 14 ] , and IEMOCAP [ 15 ] are not appropriate for this task since they are usually a fixed text corresponding to the speech with various emotions. To this end, we develop the ECD-TSE corpus through two crucial steps: Text variant generation and Emotional speech generation . Text Variant Generation: The process of text variant generation is to generate neutral text first, and then generate text variants with different emotions. First, we instruct ChatGPT-4 to generate texts with a neutral emotion. Our prompt for this step is as follows: “Please help me to build a text dataset, with each sentence consisting of 4-10 words, and in English. Please write 10 sentences, and ensure the text contains words that express neutral emotions.” Subsequently, we instruct ChatGPT-4 to modify the neutral texts by altering one or two words, thereby inducing a change in the text’s emotional connotation. Our prompt for this modification is as follows (The following example takes “sadness” as an example): “Now, I will give you a sentence. Please modify only one or two words to change the emotion to sadness. Please output only one modified sentence.” As shown in Table 1 , in this manner, we obtain texts corresponding to four additional emotional states, with only one or two words differing between the various emotions. Table 1 : Examples of text data. There are only one or two words different between different texts, but their emotional expressions are completely different. Emotion Text Neutral My sister studies in the library. Sadness My sister cries in the library. Angry My sister yells in the library.

Read the complete text