Playing with Voices: Tabletop Role-Playing Game Recordings as a Diarization Challenge
Lian Remme, Kevin Tang · 2025
Read Playing with Voices: Tabletop Role-Playing Game Recordings as a Diarization Challenge by Lian Remme, Kevin Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12714.txt.
Introduction
Lian Remme and Kevin Tang test speaker diarization on tabletop role playing game audio in two thousand twenty-five. Diarization asks who spoke when in a recording. Game players shift voices to play their characters. That built in mimicry is a natural stress test for diarizers.
The pair built a small data set of game sessions. They compare it with two known meeting corpora. Two diarizers were run on all three sets. Confusion ran higher on game audio for both tools. One tool also missed speakers by a wide count. They propose game audio as a fresh challenge set.
Catalog record
| CREATOR | Lian Remme, Kevin Tang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12714
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Playing with Voices: Tabletop Role-Playing Game Recordings as a Diarization Challenge Lian Remme, Kevin Tang Abstract This paper provides a proof of concept that audio of tabletop role-playing games (TTRPG) could serve as a challenge for diarization systems. TTRPGs are carried out mostly by conversation. Participants often alter their voices to indicate that they are talking as a fictional character. Audio processing systems are susceptible to voice conversion with or without technological assistance. TTRPG present a conversational phenomenon in which voice conversion is an inherent characteristic for an immersive gaming experience. This could make it more challenging for diarizers to pick the real speaker and determine that impersonating is just that. We present the creation of a small TTRPG audio dataset and compare it against the AMI and the ICSI corpus. The performance of two diarizers, pyannote.audio and wespeaker, were evaluated. We observed that TTRPGs’ properties result in a higher confusion rate for both diarizers. Additionally, wespeaker strongly underestimates the number of speakers in the TTRPG audio files. We propose TTRPG audio as a promising challenge for diarization systems. 1 Introduction Speaker diarization is the process of determining how many people speak in a raw audio file and who spoke in which time frames Ryant et al. 2021 ; Park et al. 2022 . Rapid improvements have been made due to recent deep learning techniques Park et al. 2022 . However, the performance of diarization systems varies depending on the domain it is applied to, and is especially bad if multiple speakers talk in a restaurant setting Ryant et al. 2021 . It has also been shown that the performance is not yet good enough for in-person role-play dialogues in health care education Medaramitta 2021 . This study proposes tabletop role-playing games (TTRPGs) as a challenge for speaker diarization systems. TTRPGs are mostly played by conversation. Multiple people pretend to be characters and either describe their characters ( descriptive ) or speak as their characters ( in-character ). During in-character conversations, people usually alter voices, e.g. by adjusting tone or speed, or even by using an accent. This property of TTRPG makes it a natural challenge for diarization. A diarizer should be able to recognize which person is speaking, even if the speaker impersonates another (probably fictional) character. We provide a proof of concept for how TTRPGs can potentially be used as an additional benchmark for a diarizer. We create a small TTRPG audio dataset, apply pyannote.audio Bredin 2023 (MIT license) and wespeaker Wang et al. 2023 (Apache-2.0) on it and compare the diarizers’ performance with their performance on the AMI dataset Kraaij et al. 2005 and the ICSI dataset Janin et al. 2003 (both CC-BY 4.0). We show that the error rate, especially the confusion about who is speaking when, is higher for the TTRPG audio than other datasets. Furthermore, we find that wespeaker underestimates the number of speaker in the TTRPG audio. 2 Background This section introduces diarization systems and how they can be fooled. We explain TTRPGs and why they could pose an interesting challenge, and we give details about the AMI and ICSI datasets. 2.1 Challenges in diarization systems There are various applications for diarization systems Nagavi et al. 2024 , such as forensic analysis Grünert et al. 2023 . Since the 1990s, there has been continuous development in diarization systems. New deep learning techniques have brought rapid improvements Park et al. 2022 . Speech diarizers are usually trained on and/or evaluated against datasets like CALLHOME Canavan et al. 1997 , the AMI corpus Kraaij et al. 2005 , the ICSI Meeting Corpus Janin et al. 2003 , the dataset of the CHiME-6 challenge Watanabe et al. 2020 , People’s Speech Galvez et al. 2021 , or VoxConverse Chung et al. 2020 . These datasets are sourced from different speech domains and differ in a wide range of settings. They are unscripted telephone conversations Canavan et al. 1997 , meeting recordings Kraaij et al. 2005 ; Janin et al. 2003 , dinner party recordings Watanabe et al. 2020 , or YouTube videos Chung et al. 2020 . Particularly notable diarizers include pyannote.audio Bredin 2023 ; Plaquet and Bredin 2023 , wespeaker Wang et al. 2023 and USTC-NELSLIP Wang et al. 2021 . Identifying which domains or speaker behaviors are challenging for an audio processing tool is an important part of making models more robust in the future. In the most recent DIHARD Diarization Challenge Ryant et al. 2021 , the best performing diarization system Wang et al. 2021 achieved a Diarization Error Rate (DER) of 19.37\% in a domain-balanced evaluation set (core evaluation set) in Track 2 (diarization from scratch). Of the 11 domains examined, the three hardiest domains were speech in restaurant by 4 to 7 speakers, web videos mostly containing multiple speakers, and meetings containing 3 to 7 speakers with a DER ranging from 35\% to 45\% Ryant et al. 2021 . This suggests that domains with three or more speakers which naturally contains more overlapping speech still pose a challenge for state-of-the-art systems. The identification and mitigation of ways to worsen an audio processing result is an active area of research, especially in the field of speaker identification and diarization. One common approach in this field is voice spoofing, which is by creating a speech sample that mimics a target speaker (see Yan et al. 2022 for an overview). Existing techniques include replaying speech samples recorded from a target speaker, speech synthesis, voice conversion, and human impersonation which is by mimicking a target speaker without technologies. Both voice professionals and non-professionals are able to spoof a system via impersonation, especially if the impersonator’s natural voice is similar to that of the target speaker Lau et al. 2004 ; Lau et al. 2005 . Speaker recognition and diarization systems also struggle with non-adversarial attacks. For instance, speech from identical twins is a phenomenon which is difficult to recognize Revathi et al. 2021 and distinguish because of similar vocal tract structure and other anatomical, physiological and physical characteristics. As even non-professionals are able to spoof speaker identification by changing their voices, speakers changing their voice to act as someone else could be a natural challenge for diarizers. This assumption is supported by the fact that diarizers’ performances are not yet good enough for in-person role-play dialogues in health care education Medaramitta 2021 . 2.2 Tabletop Role-Playing Games – TTRPGs TTRPG are mostly played out by conversation. One or more players take the role of a character living in a world created by a game master (GM). Two types of conversations can occur: In-character conversations, in which the participants talk as if they were characters, and descriptive conversations, in which the participants say what their characters are doing or what is happening in the world. Most TTRPG consist of battle scenes in which the characters fight, and role-playing scenes in which the characters do something else, e.g. talk to each other or interact with the Non-Player Characters (NPCs). These conversations exhibit interesting linguistic properties. The linguistic information can falsely indicate a change of speaker. A sentence like “I walk towards the innkeeper: ‘Could I have something to drink?’”, could be said by one person. The role