EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge
Ruskin Raj Manku, Yuzhi Tang, Xingjian Shi, Mu Li, Alex Smola · 2025
Read EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge by Ruskin Raj Manku, Yuzhi Tang, Xingjian Shi, Mu Li, Alex Smola (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23009.txt.
Description
Ruskin Raj Manku and co-authors present a 2025 benchmark for text-to-speech systems, the tools that read text aloud. Old tests miss how well a system handles feeling, odd words, or tricky phrasing. Their test set covers six hard cases: emotion, side sounds such as laughs, foreign words, complex syntax, hard items like web links and formulas, and questions. Test cases grow from human seed lines with model help.
A large audio language model grades the speech, acting as judge. It scores feeling, rhythm, tone, and pronunciation. The team ran top open and paid systems through the set, which holds 1,645 cases. Scores split the systems in fine detail and matched human taste well. The auto-built, auto-graded design lets the test grow with the tools.
Catalog record
| CREATOR | Ruskin Raj Manku, Yuzhi Tang, Xingjian Shi, Mu Li, Alex Smola |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 149 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23009