Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models

Kyowoon, Artyom, Gunu, Inchul, Jaesik · 2025

Read Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models by Kyowoon, Artyom, Gunu, Inchul, Jaesik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00832.txt.

Introduction

Kyowoon Lee and co-authors fix the delivery of synthetic speech after the fact in a 2025 paper. Text-to-speech systems now sound natural, but users still need to adjust melody and fix misread words. Older fixes bolt on special modules or retrain the model, and word fixes lean on pronunciation dictionaries that low-resource languages lack.

Their method edits the model's internal activations directly, guided by a counterfactual question: what would this layer have done for the desired output? The edit bends prosody, the rhythm and tune of speech, and corrects misread words while leaving the rest of the voice intact. It works on a pretrained model as it stands, with no extra training. Tests show the fixes land without spoiling overall quality. The door opens to studio-style touch-ups at run time.

Catalog record

CREATORKyowoon, Artyom, Gunu, Inchul, Jaesik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00832

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models Kyowoon, Artyom, Gunu, Inchul, Jaesik Abstract Recent advances in Text-to-Speech (TTS) have significantly improved speech naturalness, increasing the demand for precise prosody control and mispronunciation correction. Existing approaches for prosody manipulation often depend on specialized modules or additional training, limiting their capacity for post-hoc adjustments. Similarly, traditional mispronunciation correction relies on grapheme-to-phoneme dictionaries, making it less practical in low-resource settings. We introduce Counterfactual Activation Editing, a model-agnostic method that manipulates internal representations in a pre-trained TTS model to achieve post-hoc control of prosody and pronunciation. Experimental results show that our method effectively adjusts prosodic features and corrects mispronunciations while preserving synthesis quality. This opens the door to inference-time refinement of TTS outputs without retraining, bridging the gap between pre-trained TTS models and editable speech synthesis. Lee Stitsyuk Jho Hwang Choi KAISTSouth Korea Samsung ElectronicsSouth Korea INEEJISouth Korea keywords speech synthesis, prosody control, pronunciation control † † email: {leekwoon, stitsyuk, jaesik.choi}@kaist.ac.kr, {gunu.jho, inc.hwang}@samsung.com 1 Introduction † † footnotetext: Audio samples can be found at https://leekwoon.github.io/cae-tts Advancements in Text-to-Speech (TTS) models [ 1 , 2 , 3 , 4 , 5 ] and neural vocoders [ 6 , 7 , 8 , 9 ] have made synthetic voices nearly indistinguishable from human speech. Consequently, more attention has been attracted by refining the expressiveness of synthetic voices, particularly through the accurate manipulation of prosodic features and pronunciation. Traditionally, controlling prosody and correcting mispronunciations in TTS models have involved distinct approaches. For prosody control, one popular strategy has been learning a latent space, from which embeddings are sampled during inference to supplement the text with necessary prosodic information [ 10 , 11 , 12 , 13 ] . However, due to its unsupervised nature, selecting the right embedding can be challenging. An alternative strategy involves adding specific layers to the TTS model that are trained to extract and manipulate this prosodic information directly from the input sequence [ 4 , 5 , 14 , 15 ] , but this requires extensive preprocessing. Both of these methods, however, lack straightforward applicability to already trained models, such as Tacotron 2 [ 2 ] or Transformer TTS [ 3 ] , without undergoing a retraining process. Similarly, the field of pronunciation correction has heavily relied on pronouncing dictionaries to convert grapheme to phoneme. While effective, this method struggles with scalability, precluding TTS models from being useful in low-resource languages. Diversifying pronunciations during training helps, but the skewed word distribution in natural language (Zipfian distribution) [ 16 ] limits its utility. In general, TTS training datasets cannot match the comprehensive coverage of pronunciation dictionaries, highlighting the need for a more flexible and encompassing strategy. In this paper, we introduce an orthogonal framework leveraging Counterfactual Activation Editing (CAE) to directly manipulate the prosody and pronunciation in pretrained TTS models. Rather than adding specialized layers or relying on extensive external resources, our method exploits post-hoc decomposability by examining and editing intermediate representations after the model is trained. By posing counterfactual questions (e.g., “What would the internal representation look like if the model aimed for a higher pitch rather than the current, lower one?”), we perform fine-grained modifications to hidden activations, thereby refining prosodic features and correcting mispronunciations at inference time. The main contributions of this paper are summarized as follows: We introduce an approach that provides post-hoc adjustability of speech output, allowing for the adaptation of TTS models to new requirements without retraining. Additionally, we demonstrate the effectiveness of the proposed method through rigorous experimental validation. 2 Related Work 2.1 TTS Prosody Control Text can be spoken in various ways due to semantic nuances, speaking styles, or inherent variability. Traditional approaches, such as unit-selection, capture this variability through speech databases [ 17 ] . In contrast, recent studies model prosodic variation by predicting key prosodic features such as pitch, duration, and energy from their embedding spaces [ 4 , 5 , 14 , 18 ] , or by using implicit representations learned from a reference encoder to capture the nuanced variations not specified by text alone [ 10 , 11 , 12 ] . Controlling intermediate representations has also been considered in prior work through the use of embedding bias, calculated by assessing the extent of translation required to achieve specific modifications in acoustic features within multidimensional scaling coordinates [ 19 ] . However, such modified representations risk deviating from the data manifold, and their application has been primarily confined to controlling duration. 2.2 TTS Pronunciation Control A critical challenge for end-to-end TTS models, which operate without the aid of grapheme-to-phoneme dictionaries or predictive model, is polyphone disambiguation [ 20 ] . For TTS models to accurately convert graphemes into phonemes, their linguistic encoder must internalize the varied pronunciation rules, but fully internalizing them is difficult, leading to inevitable pronunciation errors in synthesized speech. To mitigate mispronunciations, unit selection concatenates recorded speech fragments from a database. However, this often leads to noticeable join artifacts and requires a single-speaker database, limiting the use of non-target speaker data. Recently, a model-centric approach called as the Speech Audio Corrector (SAC) [ 21 ] has been introduced. It leverages speech codes aligned with words, derived from self-supervised learning models to correct mispronunciations at the word level. In contrast, we introduce a model-agnostic approach that manipulates intermediate representations in TTS models. 3 Proposed Method (a) Layer-level Analysis (b) Neuron-level Analysis Figure 1 : Acoustic Correlation Analysis. (a) Classifier prediction losses across Tacotron 2 encoder layers, normalized with the highest value as one. (b) Effect of training only the most or least important neurons on classifier performance. We first analyze intermediate representations produced by the encoder of an end-to-end TTS model, applicable to any encoder-decoder architectures. Consider a sequence of input features \mathbf{t}=\{\mathbf{t}_{1},\dots,\mathbf{t}_{n}\} , and let \mathbb{M} be a neural network model that transforms \mathbf{t} into a series of internal representations \mathbf{t}\xrightarrow{\mathbb{M}}\mathbf{x}=\{\mathbf{x}_{1},\dots,\mathbf{x}_{n}\} . In this context, \mathbb{M} represents the encoder, with \mathbf{t} as either grapheme or phoneme embeddings. Let \mathbf{x}_{i} denote the activation in the last layer of encoder for the i -th input feature. 1 1 1 While our analysis primarily concentrates on neurons in the top layer of the encoder, the approach can be applied to other layers. 3.1 Acoustic Correlation Analysis In the context of a classification task 2 2 2 Prosody typically suits regression due to its continuous nature; however, for clarity, we discuss it under classification contexts here. , we aim to predict a specific property, denoted as \mathbf{l}_{i} , from a set of properties \mathcal{P} presumed to be inherently learned by the model \mathbb{M} , for example, prosodic or pronunciation features in TTS model. Our objective is to examine intermediate representations within \mathbb{M} to assess their significance in recognizing the property \mathbf{l}_{i}\in\mathcal{P} . For analyzing prosody properties such as duration, pitch, and energy, we first extract durations using an automatic speech recognition (ASR) system with connectionist temporal classification (CTC) alongside a backtranslation loss inspired by [ 22 ] . This system facilitates the extraction of precise alignments by sorting the posterior probabilities and applying monotonic alignment search (MAS). Subsequently, we calculate the average pitch and energy values based on these alignments to provide a comprehensive analysis. For pronunciation

Read the complete text