EmojiVoice: Towards long-term controllable expressivity in robot speech
Paige Tuttösí, Shivam Mehta, Zachary Syvenky, Bermet Burkanova, Gustav Eje Henter, and Angelica Lim · 2025
Read EmojiVoice: Towards long-term controllable expressivity in robot speech by Paige Tuttösí, Shivam Mehta, Zachary Syvenky, Bermet Burkanova, Gustav Eje Henter, and Angelica Lim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15085.txt.
Introduction
Paige Tuttösí and co-authors present EmojiVoice in this 2025 paper, a toolkit that gives social robots a more varied voice over long talks. Human speakers shift expressivity to hold a listener's interest. Robots usually get one cheerful voice and keep it flat. New speech systems mimic human expression but are too heavy to run offline on a robot.
EmojiVoice uses emojis as fine-grained prompts for tone, phrase by phrase. A light speech engine renders the voice in real time on the robot itself. The team tried it in three settings: a scripted assistant chat, a storytelling robot, and a free-talking agent. Listeners rated varied emoji prompting as more expressive over time in storytelling. In the assistant case, though, a plainer voice was preferred.
Catalog record
| CREATOR | Paige Tuttösí, Shivam Mehta, Zachary Syvenky, Bermet Burkanova, Gustav Eje Henter, and Angelica Lim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15085
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EmojiVoice: Towards long-term controllable expressivity in robot speech Paige Tuttösí, Shivam Mehta, Zachary Syvenky, Bermet Burkanova, Gustav Eje Henter, and Angelica Lim Abstract Humans vary their expressivity when speaking for extended periods to maintain engagement with their listener. Although social robots tend to be deployed with “expressive” joyful voices, they lack this long-term variation found in human speech. Foundation model text-to-speech systems are beginning to mimic the expressivity in human speech, but they are difficult to deploy offline on robots. We present EmojiVoice, a free, customizable text-to-speech (TTS) toolkit that allows social roboticists to build temporally variable, expressive speech on social robots. We introduce emoji-prompting to allow fine-grained control of expressivity on a phase level and use the lightweight Matcha-TTS backbone to generate speech in real-time. We explore three case studies: (1) a scripted conversation with a robot assistant, (2) a storytelling robot, and (3) an autonomous speech-to-speech interactive agent. We found that using varied emoji prompting improved the perception and expressivity of speech over a long period in a storytelling task, but expressive voice was not preferred in the assistant use case. I INTRODUCTION Imagine a robot telling a 10-minute story to children. How would you like the robot to speak? The expression of paralinguistics such as emotions is an integral part of human speech [ 1 ] , and humans convey expressivity by changing their expression over time [ 2 , 3 ] . Studies of expressivity in robot voices [ 4 , 5 , 6 , 7 , 8 ] use out-of-box voices such as those built into Nao and Pepper or commercial text-to-speech (TTS) such as Amazon Polly 1 1 1 https://aws.amazon.com/polly/ . These voices may contain, for example, one expressive ‘‘joyful’’ voice as an expressive voice. As a result, the use of a high-pitched and ‘‘joyful’’ voice 2 2 2 http://doc.aldebaran.com/2-4/naoqi/audio/altexttospeech-tuto.html?highlight=joyful is common when attempting to give the impression of an expressive robot [ 9 ] , mirroring engaging speakers who use a higher-than-average pitch and greater pitch range [ 10 ] . Nevertheless, major challenges remain in current TTS systems for robots to mirror human vocal expressivity. First, although “expressive” and “emotional” TTS exist, they are difficult to deploy on robots; they do not work in real-time, have limited controllability, and are often not customizable. HRI researchers are not easily able to add their own expressive styles relevant to their use cases, nor do they have fine-grained control over a voice to explore specific hypotheses. Secondly, the use of “expressive” TTS in long-term interactions, such as reading a story, is poorly understood. As such, there is a need for a synthesized voice for robots that is not only “joyful” but temporally expressive, selecting the correct expressive style given the task and the linguistic context. We must, additionally, understand how and when different expressive styles should be deployed. In this paper, we propose the use of emoji prompting to control long-term varied expressivity in robot speech (Fig. 1 ). Our contributions are as follows. • Introduce EmojiVoice, a free, open-source toolkit that enables social roboticists to create controllable, temporally variable, expressive voices for their robots using Matcha-TTS [ 11 ] . • Propose emojis as a new means to control text-to-speech. • Explore whether phrase-by-phrase emoji prompting can increase the perception of expressiveness. • Investigate in which tasks a temporally variable expressive voice (vs singularly expressive voice) is effectively deployed. Fig. 1: Miroka robot speaking with EmojiVoice expressive TTS. II RELATED WORK II-A Controlling expressivity Many expressive TTS systems are controlled through a series of complex models and hours of expressive training data [ 12 , 13 ] , which is contrary to the need for minimalist systems in robotics and is not flexible to fast fine-tuning to a specific use case. So-called “expressive” TTS systems mostly focus on increasing pitch and pitch range [ 12 , 13 ] , as previous research has found that these modifications drive expressivity in human speech [ 14 , 15 ] . Recent advancements in long-term TTS have improved smoothing over sentences by including contextual information from previous phrases [ 16 ] . However, there is a lack of work exploring how this type of expressive voice performs over long periods of speech; for example, does listening to a singularly “joyful” expressive TTS for a long time become monotonous? One of the most related studies is a recent dataset released to help train Chinese expressive storytelling TTS through multiple annotations. They include emotional colouring, with the goal of creating a “storytelling voice” [ 17 ] . They, however, do not provide an open-source model nor information on the model size, stability, and synthesis time of their baseline. Further, their use of an encoder to control emotion expressivity does not allow the user explicit expressive control, which has become a trend in TTS (especially in state-of-the-art paid solutions), where the expression is automatically selected based on text-based semantics and context 3 3 3 https://www.hume.ai/blog/octave-the-first-text-to-speech-model-that-understands-what-its-saying , https://platform.openai.com/docs/models/gpt-4o-mini-tts , https://platform.openai.com/docs/models/gpt-4o-mini-tts , and may not be appropriate for HRI studies. II-B Constraints of speech models in robotics Hardware capability continues to be a bottleneck in the explosive field of artificial intelligence (AI), where new, faster, and more efficient systems are often achieved at the expense of reliability, security, and cost [ 18 ] . Many social robots deployed in HRI studies (e.g. Nao and Pepper: Aldebaran, United Robotics Group, iCub: Istituto Italiano di Tecnologia, LOVOT: GROOVE X, Furhat: Furhat Robotics) do not have an on-board GPU or TPU. A new generation of robots with GPUs are becoming available (e.g. Reachy: Pollen robotics, Mirokaï: Enchanted Tools). Yet, less than twenty of the 263 robots available on IEEE Robots database 4 4 4 https://robotsguide.com/ specify that they are equipped with GPU hardware. As such, AI models deployed on-board robots ideally have a small footprint and require a limited amount of resources at inference time [ 19 , 20 ] . These hardware limitations are often circumvented through the use of cloud computing. Although this may be an adequate solution for in lab research deployment, often in-the-wild one cannot rely on a steady nor fast internet connection. This can lead to latency or even a complete inability to deploy interactive systems in-the-wild. In addition, the use of cloud resources can result in several security and privacy concerns [ 21 , 22 ] . Lastly, social roboticists do not always come from an engineering background and may not have the domain knowledge required to deploy free-to-use TTS systems that often have poor documentation on their local deployment and customization. Because of this, HRI studies often use out-of-box paid solutions or those running on Hugging Face Spaces, which, due to shared GPU limitations, tend to not run in real-time resulting in the need to pre-render audio files, limiting the adaptability of the system. Fig. 2: Matcha-TTS architecture [ 11 ] with the addition of emoji prompting to the text encoder and flow prediction network. TABLE I: Comparison of open source TTS systems and robot-centric features VoiceCraft [ 23 ] CoquiXTTS [ 24 ] Parler-TTS [ 25 ] FastPitch [ 26 ] Matcha-TTS Emoji-fine-tuned Matcha-TTS < 100k parameters ✗ ✗ ✗ ✓ ✓ ✓ No Hallucinations ✗ ✗