Do We Still Need Audio? Rethinking Speaker Diarization with a Text-Based Approach Using Multiple Prediction Models
Peilin Wu, Jinho D. Choi · 2025
Read Do We Still Need Audio? Rethinking Speaker Diarization with a Text-Based Approach Using Multiple Prediction Models by Peilin Wu, Jinho D. Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11344.txt.
Introduction
Peilin Wu and Jinho Choi ask whether speaker diarization even needs audio, in this 2025 paper. Diarization labels who spoke when in a conversation. Audio systems struggle with poor recordings and similar voices, so the pair try working from the transcript alone.
Their text method hunts for sentence-level speaker changes in dialogue. Two models were built: one predicts each change directly, while the other combines several predictors. On a varied set of conversation scenes, the combined model rivaled top audio systems. It did especially well on short conversations. The authors argue that meaning cues deserve a bigger role in diarization.
Catalog record
| CREATOR | Peilin Wu, Jinho D. Choi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do We Still Need Audio? Rethinking Speaker Diarization with a Text-Based Approach Using Multiple Prediction Models Peilin Wu, Jinho D. Choi Abstract We present a novel approach to Speaker Diarization (SD) by leveraging text-based methods focused on Sentence-level Speaker Change Detection within dialogues. Unlike audio-based SD systems, which are often challenged by audio quality and speaker similarity, our approach utilizes the dialogue transcript alone. Two models are developed: the Single Prediction Model (SPM) and the Multiple Prediction Model (MPM), both of which demonstrate significant improvements in identifying speaker changes, particularly in short conversations. Our findings, based on a curated dataset encompassing diverse conversational scenarios, reveal that the text-based SD approach, especially the MPM, performs competitively against state-of-the-art audio-based SD systems, with superior performance in short conversational contexts. This paper not only showcases the potential of leveraging linguistic features for SD but also highlights the importance of integrating semantic understanding into SD systems, opening avenues for future research in multimodal and semantic feature-based diarization. 1 Introduction Speaker Diarization (SD), essential for enhancing Speech-To-Text systems, identifies "who speaks what" in audio conversations jointly with Automatic Speech Recognition (ASR) systems. This technology is crucial for various recording applications and preparing AI training datasets, where distinguishing speakers enhances dialogue understanding and AI responses. SD methodologies have traditionally been categorized into two approaches: modular and end-to-end systems. The modular approach involved distinct phases of segmentation ( Bredin 2023 ; Plaquet and Bredin 2023 ), feature extraction like x-vector Snyder et al. 2018 , ECAPA-TDNN Desplanques et al. 2020 or TitaNet Koluguri et al. 2021 , and clustering methods such as spectral clustering (SC) Wang et al. 2022 and agglomerative hierarchical clustering (AHC) Pedregosa et al. 2011 , each optimized for specific aspects of the SD process. On the other hand, end-to-end systems like MSDD Park et al. 2022 or TOLD Wang et al. 2023 represented a more holistic approach, leveraging neural networks to perform diarization in an integrated manner, thereby streamlining the process and potentially enhancing accuracy. Despite all the advantages, audio-based approaches often suffered from low quality audio, similar acoustic features, and rapid change in speech causing faulty segmentation or clustering. Recent advancements have explored the integration of semantic features to improve SD results Park et al. 2023 . These attempts have ranged from utilizing text features directly Flemotomos et al. 2020 to employing such models for error correction in post-processing stages ( Paturi et al. 2023 ; Wang et al. 2024 ). Joint ASR+SD efforts Kanda et al. 2022 and multimodal approaches Cheng et al. 2023 have also been explored. However, these efforts in using semantic features have encountered several limitations: (a) The semantic features were often relied on outdated language models or were only used for a post-processing to rectify errors from preceding audio-based SD models, which did not fully exploit the semantic features. (b) There has been an absence of research exploring the use of text as the sole input for SD. This gap highlights a missed opportunity to fully explore the capabilities of semantic features in SD. In response to these challenges, this paper proposes a novel text-based SD approach leveraging only the dialogue transcript as input. Our methodology is evaluated against a curated dataset and compared with multiple recent audio-based SD models. Our findings demonstrate that while our model excels in handling short conversations, it also delivers competitive performance in longer dialogues, challenging the conventional reliance on audio features. The main contributions of this paper includes: 1. Text-based sentence-level SD approaches using text as the only input that which achieves state-of-the-art result for short conversation. 2. Data processing pipeline tailored to optimize SD on ASR-generated transcripts. 3. Comprehensive analysis about performance and error types of the text-based approach. 2 Text-based Speaker Diarization We tackle the task of text-based SD as Sentence-level Speaker Change Detection in 2-speaker conversations. This approach prioritizes sentence-level analysis over word-level for its richer contextual information, which is more conducive to accurate speaker identification. Such granularity also addresses the label imbalance issue prevalent in word-level diarization, offering a more balanced dataset for model training and evaluation. 2-speaker conversations represent a common and pragmatically significant scenario, making it an ideal focus for demonstrating the capabilities of text-based SD. Furthermore, the methodologies developed for dual-speaker contexts can be extended to multi-speaker scenarios with minimal adjustments as shown in Appendix A.4 . 2.1 Single Prediction Model The single prediction model (SPM) operates by evaluating the probability of a speaker change between sentences, using surrounding utterances as context. Let S=\{s_{1},s_{2},...,s_{n}\} be a sequence of n sentences in a conversation. The objective is to predict a binary variable y_{i} for each pair of consecutive sentences (s_{i},s_{i+1}) , where y_{i}=1 if a speaker change occurs between s_{i} and s_{i+1} , or y_{i}=0 otherwise. The change prediction result for S will be a sequence of speaker change predictions R=\{y_{1},y_{2},...,y_{n-1}\} which can be used to deduce the final speaker information. The model utilizes a context window of h sentences at front and a context window of k sentences at back of sentence s_{i+1} , where the context for the i -th prediction is defined as C_{i}=\{s_{i-h+1},s_{i-h+2},...,s_{i},s_{i+1},...,s_{i+k+1}\} . The prediction function f can be expressed as y_{i}=f(C_{i},\theta) , where \theta represents the model parameters. f can be implemented by any model that takes a sequence as an input and outputs a binary prediction label. The model is trained to minimize the loss function L , typically a binary cross-entropy loss, over all prediction points: \begin{split}L(\theta)=-\frac{1}{n-1}\sum^{n-1}_{i=1}[y_{i}\log(f(C_{i},\theta))\\ +(1-y_{i})\log(1-f(C_{i},\theta))]\end{split} (1) 2.2 Multiple Prediction Model To enhance accuracy and robustness, this paper introduces a multiple prediction model (MPM) that aggregates predictions over several windows within a dialogue so that more contextual information is leveraged on one prediction. The MPM extends the SPM by making predictions over multiple points within a sliding window across the conversation. Let W=\{w_{1},w_{2},...,w_{m}\} be a sequence of windows, where each window w_{j} consists of a subsequence of sentences from S , and m is the total number of windows covering the conversation. Each window w_{j} overlaps with its predecessors and successors. The objective is to predict a sequence of binary variables y_{i} for each window w_{j} , where each element of y_{i} corresponds to a potential speaker change within w_{j} between two consecutive sentences. The prediction for window w_{j} is given by: y_{j}=g(w_{j},\phi) , where g represents the multiple prediction function like f , and \phi denotes the model parameters specific to the multiple prediction task. The model is trained to minimize a similar loss function L^{\prime} , adjusted for multiple predictions: \begin{split}L^{\prime}(\phi)=-\frac{1}{\sum^{m}_{j=1}|w_{j}|-1}\sum^{m}_{j=1}\sum^{|w_{j}|-1}_{i=1}\\ [y_{ji}\log(g(w_{j},\phi)_{i})+(1-y_{ji})\log(1-g(w_{j},\phi)_{i})]\end{split} (2) where y_{ji} is the i -th element of y_{j} , and g(w_{j},\phi)_{i} is the i -th prediction of g for window w_{j} . To consolidate predictions across overlapping windows, an aggregation mechanism is employed. For any given potential speaker change point p , predictions from all windows encompassing p are aggregated to determine the final decision Y_{p} . Y_{p}=Aggregate(\{g(w_{j},\phi)_{p}|p\in w_{j}\}) (3) The aggregation function could be a simple majority vote, a weighted average based on confidence scores, or another suitable method designed to optimize prediction accuracy. Figure 1: Demonstration of SPM and MPM on a six-sentence conversation. 2.3 Data Processing Acknowledging the primary application of SD on ASR-generated transcripts, the