RephraseTTS: Dynamic Length Text based Speech Insertion with Speaker Style Transfer

Neeraj Matiyali, Siddharth Shrivastava, Gaurav Sharma · 2025

Read RephraseTTS: Dynamic Length Text based Speech Insertion with Speaker Style Transfer by Neeraj Matiyali, Siddharth Shrivastava, Gaurav Sharma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17031.txt.

Introduction

Matiyali, Shrivastava, and Sharma built a tool in twenty twenty-five to slip new speech into old speech when a script is fixed. The new bit must match the voice, pace, and tone of the old clip. Past text-to-speech tools can sound off when they patch a clip.

Their tool reads the full script and the part clip and fits a new bit of a length it picks on the fly. Pace of the old clip guides that length call. On a known speech set it beat a past patch tool in tests and in a user check. Voice traits and tune held well in the patch.

Catalog record

CREATORNeeraj Matiyali, Siddharth Shrivastava, Gaurav Sharma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17031

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RephraseTTS: Dynamic Length Text based Speech Insertion with Speaker Style Transfer Neeraj Matiyali, Siddharth Shrivastava, Gaurav Sharma Abstract We propose a method for the task of text-conditioned speech insertion, i.e. inserting a speech sample in an input speech sample, conditioned on the corresponding complete text transcript. An example use case of the task would be to update the speech audio when corrections are done on the corresponding text transcript. The proposed method follows a transformer-based non-autoregressive approach that allows speech insertions of variable lengths, which are dynamically determined during inference, based on the text transcript and tempo of the available partial input. It is capable of maintaining the speaker’s voice characteristics, prosody and other spectral properties of the available speech input. Results from our experiments and user study on LibriTTS show that our method outperforms baselines based on an existing adaptive text to speech method. We also provide numerous qualitative results to appreciate the quality of the output from the proposed method. 1 Introduction Large amount of audio data is being created and consumed every minute for a variety of pursuits. Yet processing audio is still quite hard and time consuming, e.g . if a single mistake is made during recording, the user is forced to rerecord the complete segment. A potential solution to this problem is to be able to use the corresponding text transcript to manipulate the audio data. If a change is needed, the user could change corresponding part of the text and an audio inpainting method could automatically make the corresponding change in the audio signal. In this paper, we address this problem and propose a network which can transfer speaker style and add or replace existing speech segments. Earlier works in this area take an audio and a text transcript as input. The text transcript generally has a few words added or replaced as compared to the content of the input audio. The output is an audio for the text transcript in the style of the speaker from the input audio. There are two settings for this problem, (i) replace a segment of speech with an audio of equal length ( Borsos, Sharifi, and Tagliasacchi 2022 ) i.e . the length of the replaced audio fragment and hence the overall audio remains the same, (ii) insert dynamic length audio segments ( Tang et al. 2021 ; Yin et al. 2022 ) , thus increasing or decreasing the size of the output audio. Our proposed work belongs to the latter area. This problem is very challenging as it requires the system to automatically align the phonemes across text and speech, and does not place any restrictions on the length of the hole to be replaced — the user can potentially replace a single word with multiple words of longer duration. The inserted speech in the present case may be of different length than what was present in the original sample, e.g . in an original sample of “Fred was present in the meeting”, the user might want to replace “Fred” with “Fred Flintstone”. In addition to the dynamic length of the part to be inserted, further challenges come in recreating the speaker and speech characteristics in the inpainted parts. Here, the problem bears similarities to adaptive text to speech problem ( Choi et al. 2020 ; Casanova et al. 2021 ; Min et al. 2021 ) , where a small reference sample of speech for a particular speaker is given and the task is to perform text to speech with new text segments in the style of the reference speech. In the speech insertion problem the context around the hole to be inserted provides the speaker characteristics. The task then is not only to generate a plausible speech segment for the replaced text, but also to match the context in terms of tempo, prosody and other higher level speaker characteristics of speech. The proposed method is designed to incorporate such aspects. Our proposed network is based on FastSpeech2 ( Ren et al. 2020 ) , and adds an audio stream in parallel to the phoneme stream. The audio encoder processes the input audio context in form of mel-spectrograms where frames to be inserted are removed. We also propose a cross-modal attention module for extracting speech characteristics from the audio stream and using them to enhance the phoneme representation. We train the network using local and global adversarial losses with multi-layer discriminator feature matching and a style matching loss to improve the quality of synthesized speech samples. Similar to ( Borsos, Sharifi, and Tagliasacchi 2022 ; Yin et al. 2022 ) , we derive a speech insertion baseline based on one of the state of the art Adaptive TTS methods Meta-StyleSpeech ( Min et al. 2021 ) , and compare our proposed method against it. Our main contributions are as follows. (i) We propose a novel deep neural network for dynamic length speech insertion using cross-modal attention with adversarial local, global and style losses. (ii) We perform an exhaustive user study of the output generated by the proposed network on a public dataset. (iii) We empirically evaluate and perform ablation studies to highlight the effectiveness of the proposed method. (iv) We provide several qualitative results (in supplementary) to demonstrate the effectiveness of our method c.f . compared methods. 2 Related Work 2.1 Text-Conditioned Speech Inpainting Text-conditioned speech inpainting was first proposed by ( Prablanc et al. 2016 ) . Their method first synthesizes speech from text and then uses voice conversion mapping on the synthesized speech to match the style of the observed speech. A disadvantage of this approach is that it requires a significant amount of data for each target speaker for learning the speaker specific conversion mappings. Recently, ( Borsos, Sharifi, and Tagliasacchi 2022 ) have proposed a method for the text-conditioned speech inpainting, which is based on multi-modal network Perceiver IO ( Jaegle et al. 2021 ) . Similar to our method, their method does not require any additional data other than the observed parts of the input speech. However, their setup of text-conditioned speech inpainting has a disadvantage—a fixed mask has to be provided in the input and it can only inpaint speech which has the same duration as the mask. In contrast, in our setup, our model automatically infers the duration of the gap that is to be inpainted based on the text transcript containing the replacement text. 2.2 Text-to-Speech Synthesis Advances in neural text-to-Speech synthesis ( Shen et al. 2018 ; Ren et al. 2020 ) have shown the capability of synthesizing natural speech free of artifacts. FastSpeech2 ( Ren et al. 2020 ) has shown significant speedup in synthesis of speech samples by using non-autoregressive decoding of speech. Our method follows the design of FastSpeech2 to leverage such advantages. 2.3 Adaptive Multi-Speaker TTS A closely related problem to text-conditioned speech inpainting is adaptive multi-speaker TTS (also known as voice cloning) ( Arik et al. 2018 ; Chen et al. 2018 ; Jia et al. 2018 ) . Recent advances in adaptive multi-speaker TTS ( Choi et al. 2020 ; Casanova et al. 2021 ; Min et al. 2021 ) have made it possible for text-to-speech systems to synthesize speech in

Read the complete text