Multi-interaction TTS toward professional recording reproduction

Hiroki, Kenichi, Aya, Yusuke · 2025

Read Multi-interaction TTS toward professional recording reproduction by Hiroki, Kenichi, Aya, Yusuke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00808.txt.

Introduction

A team of NTT researchers with first names Hiroki, Kenichi, Aya, and Yusuke propose a speech synthesis system that takes direction like a voice actor does. In professional recording, a director gives feedback take after take until the delivery matches the intended style. Standard text-to-speech systems cannot do that: once speech is generated, its style is hard to nudge. Writing in twenty twenty-five, the authors model the give and take between a director and an actor and let users refine synthesized speech across several rounds of plain feedback.

Experiments with their model and a matching data set show that each round of direction moves the voice closer to what the user asked for. Style changes follow the feedback rather than drifting at random. The authors present multi-round interaction as a practical step toward synthetic voices that can be rehearsed and corrected, much like human performers.

Catalog record

CREATORHiroki, Kenichi, Aya, Yusuke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00808

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-interaction TTS toward professional recording reproduction Hiroki, Kenichi, Aya, Yusuke Abstract Voice directors often iteratively refine voice actors’ performances by providing feedback to achieve the desired outcome. While this iterative feedback-based refinement process is important in actual recordings, it has been overlooked in text-to-speech synthesis (TTS). As a result, fine-grained style refinement after the initial synthesis is not possible, even though the synthesized speech often deviates from the user’s intended style. To address this issue, we propose a TTS method with multi-step interaction that allows users to intuitively and rapidly refine synthesized speech. Our approach models the interaction between the TTS model and its user to emulate the relationship between voice actors and voice directors. Experiments show that the proposed model with its corresponding dataset enables iterative style refinements in accordance with users’ directions, thus demonstrating its multi-interaction capability. Kanagawa Fujita Watanabe Ijima nocounter]NTT, Inc.Japan keywords text-to-speech, expressive speech, speaking style refinement † † email: [email protected], [email protected] 1 Introduction Almost all creative activities entail a lot of trial and error to produce high-quality work [ 1 , 2 , 3 ] . Film or television directors often require multiple takes, while photographers frequently adjust angles and camera settings. Similarly, voice directors often iteratively refine voice actors’ performance by providing feedback to achieve the desired speaking style (see Fig. 1(a) ) as in performance refinement of actors [ 4 , 5 ] . Like the manual creation process, recent machine learning tasks now include iterative interactions between users and systems; examples include interactive image generation [ 6 , 7 , 8 ] and program coding [ 9 , 10 ] . This trend is being further accelerated by the emergence of large language models (LLMs) [ 11 , 12 ] . However, in speech synthesis, these iterative refinement processes have not been examined. Recent text-to-speech synthesis (TTS) has advanced rapidly, achieving human-level speech synthesis with naturalness comparable to that of recorded speech [ 13 , 14 ] . In addition to naturalness, expressiveness has improved significantly, enabling the reproduction of diverse speaking styles. Research into expressive TTS divides speech into three elements: text, speaker, and style information. Starting from a simple style-ID based approach [ 15 ] and the methods replacing pitch and durations from original speech [ 16 , 17 ] , style information has recently been modeled using data-driven techniques with speech embedding [ 18 , 19 , 20 , 21 ] . Due to recent advances in LLMs, style information can be usefully inferred from text descriptions of speaking styles [ 22 , 23 , 24 ] . However, these methods do not support feedback from previously generated speech, requiring TTS users to iteratively select speech prompts or text prompts until the desired synthetic style is achieved. (a) (b) Figure 1 : Overview of the task and our proposed method. To enhance the applicability of speech synthesis by enabling the easy generation of speech with the desired style, we believe incorporating iterative refinement is promising. Building on this idea, we propose a new expressive TTS method guided by multiple directions (see Fig. 1(b) ). To incrementally refine TTS output towards the desired style, we treat the TTS model as a voice actor—akin to a voice actor converting text into speech—and design it to accept additional textual directions. The speech synthesized using the provided direction is passed to the subsequent interaction. Repeating this interaction efficiently, the style of the synthesized speech is refined. To train our model, we designed a new dataset that replicates actual recordings. This dataset includes multiple direction texts and voice actor recordings to simulate the process of style refinement. These direction texts also includes not only manipulation of speaking style but also linguistic information (e.g., the placement of silent pause) unlike the similar text-prompted-TTS or voice conversion (VC) approaches [ 25 ] . During data collection, to minimize dependency on the voice director’s skill and personality, the direction texts were prepared in advance rather than being given on the spot. As a first step toward enabling human-like interactive iterative style refinement in expressive TTS, we conducted experiments focusing only on speech embedding manipulation guided by textual direction. The results confirmed that our proposed method could approximately incorporate user’s intended directions in the synthesized speech without degrading naturalness. Moreover, our case analysis also highlighted remaining issues such as 1) specific position in a sentence and 2) dealing with ambiguous expressions, providing important insights for further style refinement improvement 1 1 1 https://ntt-hilab-gensp.github.io/ssw13multiinteractiontts/ . 2 Proposed method To model the interaction between the voice director and the voice actor, we constructed a dataset that imitates an actual recording by them. During the recording of this dataset, a director iteratively gave acting directions, and the voice actor then reflected the given directions. To model this direction cycle, we constructed a model capable of generating refined speech that reflects the given directions. We would like to emphasize that style refinement is intended to include linguistic refinement in the future. Therefore, our task is entirely different from voice conversion using text prompts [ 25 ] , in which only the speaker characteristics are changed while preserving the linguistic information. This section explains the dataset (Fig. 2 ), backbone TTS model, and style refiner (Fig. 3 ). Figure 2 : Overview of the direction cycle in dataset collection. Table 1 : Example of style label, manuscript, and directions translated from Japanese. Example Style Label Resignedly Manuscript I don’t have time, okay? It’s not like I wanted this either… Direction 1 Say it in a more restrained tone, with a sigh mixed in. Keep the delivery flat and steady, with little variation in strength. Direction 2 Let’s try speaking so that your intonation gently falls at the end without sharp breaks, letting your words flow smoothly. Keep a slightly laid-back tone in mind. Direction 3 Place just a slight pause between words, and speak in a tone that conveys resignation rather than urgency; overall, keep your pitch low. 2.1 Dataset To model interactions between the voice director and the voice actor, we constructed a dataset containing both speeches iteratively refined by the voice actor, and directions provided by the voice director (Fig. 2 ). In actual voice recording sessions, voice directors provide acting instruction to voice actors in various ways. Such instruction can be categorized as follows: 1. Instructions for modifying para-linguistic and non-linguistic information in the acted speech (e.g., “Speak more brightly,” “Be more considerate of the other person’s feelings”). 2. Instructions for modifying linguistic information (e.g., “Insert a silent-pause after this word,” “Change the location of stress (or accent (for pitch-accent language, i.e., Japanese))”). 3. Demonstrative instructions, where the director provides a performance example (e.g., “Follow my example”). 4. Instructions involving gestures and body language. While it would be ideal to construct a dataset that contains all of these categories, creating such a comprehensive dataset is highly complex and presents substantial challenges. Therefore, in this study, we constructed a dataset limited to categories 1 and 2, which can be expressed solely in textual form. Furthermore, in most actual voice recording sessions, voice directors iteratively review the results and provide instructions to guide voice actors toward the desired outcome. However, replicating

Read the complete text