Rasmalai : R esources for A daptive S peech M odeling in Indi A n L anguages with A ccents and I ntonations

Ashwin, Yoach, Sherry, Praveen, Sanchit, Mitesh · 2025

Read Rasmalai : R esources for A daptive S peech M odeling in Indi A n L anguages with A ccents and I ntonations by Ashwin, Yoach, Sherry, Praveen, Sanchit, Mitesh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18609.txt.

Introduction

Ashwin Sankar and co-authors release a giant speech dataset for Indian languages in a 2025 paper. Their collection covers twenty-three Indian languages plus English, with thousands of hours of speech. Rich text descriptions tag each clip with speaker, accent, emotion, style, and background conditions, drawn from tens of millions of annotations. The aim is speech synthesis that can be steered in fine detail.

Using the data, they built an open text-to-speech system guided by those descriptions. Tests showed it can copy named speakers, follow style descriptions, and hit requested traits accurately. Expressive traits also transferred within and across languages. The authors claim a new standard for this task.

Catalog record

CREATORAshwin, Yoach, Sherry, Praveen, Sanchit, Mitesh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18609

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rasmalai : R esources for A daptive S peech M odeling in Indi A n L anguages with A ccents and I ntonations Ashwin, Yoach, Sherry, Praveen, Sanchit, Mitesh Abstract We introduce Rasmalai , a large-scale speech dataset with rich text descriptions, designed to advance controllable and expressive text-to-speech (TTS) synthesis for 23 Indian languages and English. It comprises 13,000 hours of speech and 24 million text-description annotations with fine-grained attributes like speaker identity, accent, emotion, style, and background conditions. Using Rasmalai , we develop IndicParlerTTS , the first open-source, text-description-guided TTS for Indian languages. Systematic evaluation demonstrates its ability to generate high-quality speech for named speakers, reliably follow text descriptions and accurately synthesize specified attributes. Additionally, it effectively transfers expressive characteristics both within and across languages. IndicParlerTTS consistently achieves strong performance across these evaluations, setting a new standard for controllable multilingual expressive speech synthesis in Indian languages. Sankar Lacombe Thomas Srinivasa Varadhan Gandhi M Khapra Indian Institute of Technology MadrasIndia HuggingfaceFrance keywords speech synthesis, text-prompted, expressive 1 Introduction Text-to-Speech (TTS) synthesis has evolved beyond generating speech from a fixed set of voices using high-quality recordings, with recent advancements exploring speech- and text-prompted TTS systems. Speech-prompted TTS [ 1 , 2 , 3 , 4 ] synthesizes speech by replicating the style and speaker characteristics from a provided audio prompt. In contrast, text-prompted TTS [ 5 , 6 , 7 , 8 , 9 ] generates speech based on textual descriptions that can specify various attributes such as pitch, speed, noise levels, gender, and emotional expression. It is more user-friendly, as it removes the dependency on the availability of speech samples, allowing users to specify their desired speech characteristics via text. Training such models requires datasets containing triplets of { audio , transcript , description }, where the description guides the synthesis model to produce controlled and expressive speech with the specified attributes. Much of the progress in text-prompted TTS has been limited to English and a few high-resource languages, while Indian languages have lagged behind due to the lack of suitable datasets. While several efforts [ 10 , 11 , 12 ] have introduced studio-recorded TTS datasets, their scale remains limited. Some TTS datasets [ 11 ] provide detailed annotations for emotions and speaking styles, while others [ 13 ] , though not recorded in controlled environments, offer greater scale, speaker diversity, and broader linguistic and acoustic coverage. These datasets vary in size, quality, and style diversity but none provide structured text descriptions that explicitly characterize speaker voice, style, environment and expressive attributes. This gap highlights the need for a dataset that integrates linguistic diversity, large-scale speech data, and rich textual descriptions to enable text-prompted TTS for Indian languages. To address this gap, we propose generating text-based speech descriptions using Large Language Models (LLMs) by leveraging attribute information in the form of structured tags. Specifically, we aggregate multiple existing TTS datasets containing { audio , transcript } pairs and extract three categories of attributes: (i) metadata attributes such as age, gender, and speaker identity, where available; (ii) stylistic attributes, including Ekman emotions when annotated; and (iii) acoustic features such as pitch [ 14 ] , speaking rate, and clarity index [ 15 , 16 ] , derived using speech processing libraries. These attributes are then provided as input to an LLM to generate detailed text descriptions, creating multiple text prompts per utterance. This methodology is applied to a diverse set of { audio , transcript } datasets spanning multiple Indic languages as well as English with varied accents. The resulting dataset, Rasmalai , comprises over 13,000 hours of transcribed speech, with each utterance annotated with three distinct text-based descriptions of varying specificity and robustness (see Table 1 ). Additionally, these descriptions are translated into native languages using automated translation techniques [ 17 ] , further enhancing the dataset’s inclusivity for multilingual text-prompted TTS. Table 1 : Example showcasing (a) Descriptive, (b) Concise, and (c) Attribute Robust captions, for a set of speech attributes. Attributes Captions Jaya, female speaker, Slightly close sounding, High pitch, Expressive tone, Slightly fast pace, Great speech quality, Speaking style is Anger (a) Jaya, a female speaker, delivers speech in a slightly roomy environment with a high-pitched, expressive tone. She speaks at a slightly fast pace, with excellent overall speech quality. The intended style is anger. (b) Jaya, a female speaker, delivers high-pitched, expressive speech in a slightly enclosed environment at a fast pace, with excellent quality and an angry tone. (c) Jaya’s angry tone, with a sharp voice, echoes with exceptional quality in a moderately reverberant environment. Using Rasmalai , we train IndicParlerTTS , the first multilingual text-prompted TTS system covering 24 Indian languages. We thoroughly evaluate its performance through multiple systematic assessments. First, we assess its ability to function as a general high-quality TTS system by synthesizing natural speech in the voice of named high-quality speakers, as specified in the text prompts. Using MUSHRA tests, we find that the synthesized speech exhibits high naturalness and strong fidelity to the intended speaker. Second, we evaluate the model’s instruction-following capability by determining whether the generated speech accurately reflects the descriptions provided in the text prompts. Our results show that IndicParlerTTS effectively adheres to the specified characteristics. Third, we examine the model’s ability to generate expressive speech for speakers whose training data includes expressive variations. Our analysis shows that the model successfully synthesizes the intended emotions with very high accuracy. Fourth, we analyze the model’s capacity to synthesize expressive speech for speakers within a language even when no expressive data for that speaker was seen during training. We find that the model generalizes well in such cases, successfully performing style transfer to match the styles specified in the descriptions. Finally, we test its cross-lingual generalization capability by evaluating if the model can synthesize expressive speech in language L_{2} for a speaker in language L_{1} , even in cases where no expressive speech data was available for that speaker in either language. IndicParlerTTS consistently achieves strong performance across all these evaluations, demonstrating its robustness and adaptability. To support further research in multilingual text-prompted TTS, we will release all models and code to the community. 2 Rasmalai : An Annotated Corpus for Controllable Multilingual TTS Below we describe (i) existing TTS datasets from which {audio, text} pairs were collated (ii) our approach for generating text descriptions for these audios and (iii) statistics of our dataset. 2.1 Collating existing TTS datasets We first collate { audio , text } pairs from multiple existing Indian and English language datasets, as mentioned below: RASA [ 11 ] . A diverse studio-quality speech dataset that includes neutral readings, expressive speech with six basic Ekman emotions, and command-based interactions from platforms such as Alexa, UMANG, and DigiPay. It also includes spontaneous conversations, news readings, and audiobook narrations. Each utterance is annotated with style, emotion labels, and speaker identity. It comprises 20 speakers across 13 languages, totaling around 400 hours of speech. IndicVoices [ 18 ] . A large-scale Indian speech dataset with over 7,200 hours of read, extempore, and conversational speech from 16,237 speakers across 22 languages. Originally designed for training ASR systems, it captures diverse acoustic, linguistic, and stylistic variations across various domains

Read the complete text