A Data-Centric Framework for Addressing Phonetic and Prosodic Challenges in Russian Speech Generative Models

Kirill Borodin, Nikita Vasiliev, Vasiliy Kudryavtsev, Maxim Maslov, Mikhail Gorodnichev, Oleg Rogov, Grach Mkrtchian · 2025

Read A Data-Centric Framework for Addressing Phonetic and Prosodic Challenges in Russian Speech Generative Models by Kirill Borodin, Nikita Vasiliev, Vasiliy Kudryavtsev, Maxim Maslov, Mikhail Gorodnichev, Oleg Rogov, Grach Mkrtchian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13563.txt.

Introduction

Kirill Borodin and co-authors present Balalaika, an open pipeline that prepares Russian speech data with rich labels, in this 2025 paper. Speech models need large, clean corpora, and Russian data with stress and sound detail is scarce.

The pipeline segments audio while keeping context, merges several speech recognizers by consensus vote, and keeps word timings on request. Filters drop low-quality and mixed-speaker clips. Text gains restored punctuation, word stress marks, and phoneme forms. With it the team built a 5,100-hour corpus from many sources. Training tests showed steady gains for speech cleanup and synthesis at equal budgets.

Catalog record

CREATORKirill Borodin, Nikita Vasiliev, Vasiliy Kudryavtsev, Maxim Maslov, Mikhail Gorodnichev, Oleg Rogov, Grach Mkrtchian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Data-Centric Framework for Addressing Phonetic and Prosodic Challenges in Russian Speech Generative Models Kirill Borodin, Nikita Vasiliev, Vasiliy Kudryavtsev, Maxim Maslov, Mikhail Gorodnichev, Oleg Rogov, Grach Mkrtchian Abstract This work is still in progress Russian speech synthesis presents distinctive challenges, including vowel reduction, consonant devoicing, variable stress patterns, homograph ambiguity, and unnatural intonation. This paper introduces Balalaika, a novel dataset comprising more than 2,000 hours of studio-quality Russian speech with comprehensive textual annotations, including punctuation and stress markings. Experimental results show that models trained on Balalaika significantly outperform those trained on existing datasets in both speech synthesis and enhancement tasks. We detail the dataset construction pipeline, annotation methodology, and results of comparative evaluations. † † paperid: 3450 Version notice (5 May 2025). This preprint is an earlier version of our work and is provided solely to solicit feedback from the community. A revised manuscript has been submitted to an anonymous review venue and will be released after the end of the review period in accordance with conference policies. The code in the repository may differ from the current preprint version. 1 Introduction 1.1 Russian Language Challenges for TTS Text-to-speech (TTS) synthesis have advanced dramatically in recent years, enabling machines to produce intelligible and natural-sounding speech from written text. These systems are now widely used in applications such as virtual assistants, audio books and accessibility tools. In addition to TTS, generative models are also used in speech denoising, enhancement, and restoration, where they improve the intelligibility and quality of audio by reducing background noise or reconstructing missing segments. Together, these technologies are changing the way people interact with digital devices, making information more accessible and communication more fluent in a variety of environments and use cases. Speech synthesis in the Russian language faces unique challenges due to its complex phonetics, morphology, and syntax. These linguistic features require a more meticulous approach to the dataset that will be used by the model. The utilization of raw text–audio pairs alone is insufficient for achieving high-quality speech synthesis. First of all, the TTS of the Russian language faces phonetic problems. Specifically, the Russian language possesses a rich system of consonants [ Trubach et al.(2023)Trubach, Gorshkova, and Sklyar ] , comprising 35 phonemes, including challenging to synthesize hisses([ s],[ z]), whistles([s],[z]), and affricates([ts],[t]). Their articulation necessitates precise regulation of formants and durations to avoid distortions, such as the metallic sound characteristic of hisses and whistles. Secondly, words in Russian frequently contain clusters of consonant letters([vzgl j at] - glance, [zdrafstvUjt j E] - hello) [ Trubach et al.(2023)Trubach, Gorshkova, and Sklyar ] , which can disrupt smooth transitions between sounds. In contrast to English, Russian is characterized by a free word order [ Rodionova(2001) ] , yet for a more natural pronunciation, some words require lexical stress. Models trained with a fixed order may disregard this feature, resulting in a ”non-native” intonation. Next, we address four issues that can be resolved by effective data preprocessing. The initial issue is reduction of vowels [ Trubach et al.(2023)Trubach, Gorshkova, and Sklyar ] , a process in which vowels in stress-free syllables that undergo a change in pronunciation(spelled – [moloko], pronounced - [m5l5ko], means - milk), which is critical for natural speech. When training models on data that do not consider reduction, the resulting models produce hypercorrections, but not natural-sounding results. A similar phenomenon occurs in consonant devoicing [ Trubach et al.(2023)Trubach, Gorshkova, and Sklyar ] , where a voiced sound transitions into a voiceless one(spelled - [dub], pronounced - [dup], means - oak). The second issue to be addressed is that of mobile stresses [ Petrov(2025) ] . As stresses are not fixed, they have the capacity to modify the lexical([zamk] - [zmok], lock – castle) and grammatical meaning of a word([glva] - [golv1], head – heads). Some stresses do not obey the rules, making the use of contextual LLM-based morphological analyzers necessary. In such cases, the use of rules and dictionaries becomes ineffective. The third problem is related to the normalization of texts, which brings syntactic and prosodic challenges. Firstly, morphology is filled with 6 cases, 3 genera and 2 numerals [ Rozovskaya and Roth(2019) ] . Numerals and compound words become difficult to convert into text, as this step would require a full syntactic analysis to determine the morphological properties of the word and its relations within the sentence. Secondly, the huge number of abbreviations and acronyms require a deep semantic knowledge of the Russian language. Thirdly, due to intercultural interaction, the language is endowed with a large number of borrowed words, mainly from English, which causes a lack of consistency with the rules of the language. The fourth problem is the monotony of recordings. The most straightforward method of obtaining speech synthesis datasets is from audiobooks; however, this approach can degrade TTS quality, as intonations characteristic of audiobook narration are distinct from natural human speech [ Zhang et al.(2023)Zhang, Yeh, Beckman, Raitio, Rasipuram, Golipour, and Winarsky ] , compounding the synthesis issues. When using authentic spontaneous speech, transcription is necessary. A significant challenge at this stage is the complex punctuation. High-quality punctuation has a notable impact on the intonational and prosodic components of the generated speech. 1.2 Motivation Despite rapid advancements in multispeaker speech synthesis systems, these systems continue to suffer from the fundamental limitations outlined in Section 1.1 . Furthermore, the pre-training of several models becomes challenging due to the insufficiency of their Russian vocabularies, lacking stress information, which is a crucial component in speech synthesis. Figure 1: Visualization of the proposed annotation process. The training of custom models poses significant challenges, as there is a lack of publicly available datasets comprising hundreds of hours of high-quality live speech data with comprehensive annotations. Addressing these challenges, we have initiated the construction of a high-quality Russian speech dataset, aiming to enhance the quality of our datasets and to advance the field. 1.3 Key Contributions In this paper, we present Balalaika, a dataset collected from open sources, consisting of high-quality studio recordings of conversational Russian speech. The dataset was annotated using state-of-the-art models, resulting in high annotation accuracy and consistency, making it suitable for training speech synthesis and other generative systems. The dataset is available for personal, non-commercial use 1 1 1 https://github.com/mtuciru/balalaika . The paper is organized as follows: the second section describes our data preprocessing pipeline in detail, the third section describes the evaluation items and metrics, the fourth section discusses the results, the fifth defines the limitations of our work and the sixth section concludes. 2 Dataset Construction Pipeline 2.1 Data collection strategy The first and most important step is to determine the method of data acquisition. For this purpose it was decided to use the service Yandex Music . The albums with podcasts were used to form the dataset, since this type of compilations contains high-quality conversational speech. In terms of quality, the albums were selected with the voice recording corresponding to a studio-quality recording with minimal noise, reverberation, and other external sounds. A key criterion was the naturalness of intonation, as the dataset consists primarily of spoken speech, which addresses one of the issues discussed in Sec. 1.1 . From each source, three randomly selected recordings were reviewed, and

Read the complete text