Rhythm Features for Speaker Identification

Nick, Thomas, Dani, Shri · 2025

Read Rhythm Features for Speaker Identification by Nick, Thomas, Dani, Shri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06834.txt.

Introduction

Four authors who sign only first names, Nick, Thomas, Dani, and Shri, test whether the rhythm of speech can identify a speaker, in a 2025 paper. Rhythm means the timing pattern of sounds and syllables as a person talks. Each person has habits of timing that may act like a voice fingerprint.

The team used deep learning tools to name speakers from rhythm features alone, without using the words spoken. Their results back the idea that rhythm carries useful identity clues. But they also found a catch. When people speak in a loose, unplanned way, their rhythm shifts a lot from one time to the next. That shift makes rhythm less reliable on its own for naming a speaker.

Catalog record

CREATORNick, Thomas, Dani, Shri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06834

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rhythm Features for Speaker Identification Nick, Thomas, Dani, Shri Abstract While deep learning models have demonstrated robust performance in speaker recognition tasks, they primarily rely on low-level audio features learned empirically from spectrograms or raw waveforms. However, prior work has indicated that idiosyncratic speaking styles heavily influence the temporal structure of linguistic units in speech signals (rhythm). This makes rhythm a strong yet largely overlooked candidate for a speech identity feature. In this paper, we test this hypothesis by applying deep learning methods to perform text-independent speaker identification from rhythm features. Our findings support the usefulness of rhythmic information for speaker recognition tasks but also suggest that high intra-subject variability in ad-hoc speech can degrade its effectiveness. Mehlman Thebaud Byrd Narayanan University of Southern CaliforniaUSA Johns Hopkins UniversityMaryland, USA University of Southern CaliforniaUSA keywords speaker identification, speech rhythm, deep learning † † email: [email protected] 1 Introduction In addition to its linguistic content, human speech contains rich information about the speaker’s identity [ 1 ] . Attributes such as pitch (i.e., f_{0} ), spectral energy distribution, and amplitude envelope are highly individualistic and hence can act as useful discriminative features for determining who is speaking (speaker identification, SI) or verifying the identity of a given speaker (speaker verification, SV). However, far less attention has been given to the use of rhythm and other prosodic information, particularly the temporal structure of linguistic units such as phonemes or syllables. This is despite the fact that the rhythmic aspects of speech have been shown to be highly idiosyncratic (e.g., [ 2 , 3 ] ) and dependent predominantly on the speaker [ 4 ] . Additionally, unlike lower-level audio features, rhythm information is more robust to changes in the channel (e.g., noise, filtering) [ 5 ] and even deliberate attempts to obfuscate identity information (e.g., pitch shifting) [ 6 ] . While several prior works [ 7 , 8 ] have attempted to leverage rhythm for SI and SV, most of these methods employ classical learning methods such as hidden Markov models (HMMs). Few attempts have been made to apply more contemporary deep neural networks (DNNs) despite their widespread proliferation and state-of-the-art performance. In this paper, we explore the utility of rhythm features for DNN-based SI. Our approach uses a transformer model trained on aligned transcripts that are generated automatically from an ASR model. This makes our method text-independent, allowing it to be applied to unlabeled speech signals. We evaluate our method on LibriSpeech [ 9 ] and VoxCeleb 1 [ 10 ] , two popular speech datasets. In addition to the case in which rhythm is used in isolation, we also evaluate its use as an additional feature set to improve the performance of conventional (audio-based) SI models. 2 Background A variety of prior works have demonstrated that the rhythmic and prosodic aspects of speech convey substantial information about the speaker’s identity. For example, [ 11 ] demonstrated that human listeners were able to identify familiar speakers based on only a sinusoidal encoding of the prosodic information in speech. Phoneme durations have been shown to be heavily speaker-dependent by both [ 2 ] and [ 3 ] . Additionally, both [ 12 ] and [ 4 ] observed that these prosodic variations tend to be fairly stable for a given speaker, regardless of the text that is being spoken. Other works, however, such as [ 13 ] have found that content can have a substantial influence on certain aspects of the speaker’s prosody. Based on these observations, a few attempts have been made to leverage rhythmic features to improve the performance of classical speaker recognition methods (e.g., HMMs). For example, [ 14 ] proposed normalizing phoneme durations to improve the robustness of HMM-based verification systems. However, they found that this produced only relatively small improvements. Meanwhile, the authors in [ 7 ] explicitly used context-dependent phoneme durations for SV. They demonstrated that using these features in conjunction with traditional audio-based ones slightly improved the equal-error rate (EER) compared to audio features alone. Meanwhile, a set of text-independent prosodic features (phone duration, f_{0} , and energy) were individually tested for speaker recognition in [ 8 ] and found to carry decent discriminatory power. Despite these successes in classical systems, more contemporary SI methods have largely ignored rhythmic information, opting instead to learn empirically derived features directly from the speech signal. Phoneme durations were, however, used to train a transformer-based speaker embedding model in [ 15 ] . The authors demonstrate that these speaker embeddings produced fairly robust performance in a speaker verification task ( 10\% EER vs. 2\% for x-vectors) and also improved speech synthesis quality. However, their approach used manually aligned annotation, limiting its utility for real-world speech data. Meanwhile, [ 6 ] used mean rhythm durations (computed using an HMM-GMM model) as features to attack DNN-based voice anonymization systems. While their approach succeeds in reducing the EER of the anonymized speech, it does not account for potentially context-dependent rhythmic aspects. It also does not directly apply any deep learning techniques to the durational features. 3 Method Figure 1: Overviews of the proposed rhythm-based speaker identification framework. Figure 1 summarizes our approach to rhythm-based SI. First, we use the pre-trained WhisperX model [ 16 ] to extract time-aligned transcripts from the audio file. These transcripts are then converted into a frame-aligned character sequence (FACS) with one character per frame and repetition to represent character duration. These FACS are then used to train a transformer encoder model to predict speaker identity. We outline each of these steps in greater detail in the following sections. 3.1 FACS Extraction Text FACS Representation ‘he spoke the last two words’ hhee**sssppppookee*thee*llaassstt***twwwoo***wwwwworrrddddss ‘in these assemblies we ought to’ inn***thhhessee***assssssssseemmmbbbllliieess****wwweee*****ouugghht**ttoo ‘she stepped boldly into the room’ shhee***ssttepppppedd***bbbboollldddddllllyy*iiiiiiiiiiiiiinnnnttooo*thee**rrooooom Table 1: FACS representation of different spoken phrases. The null character (which indicated no speech for a given frame) is represented by ‘*’. WhisperX extends the capabilities of the robust Whisper [ 17 ] ASR model to generate transcriptions that are time-aligned. To produce our FACS features, we applied WhisperX 1 1 1 We used the pretrained implementation available at https://github.com/m-bain/whisperX/ to the speech data to obtain the predicted transcript along with character-level time stamps. These were then converted into a frame-by-frame sequence of characters, with one character per 20 ms frame. Each character was repeated for the number of frames it was aligned to, thereby capturing the underlying temporal structure of speech. For non-speech frames, we assigned a unique null character to also capture the prosodic information conveyed by pauses and gaps. Note that, unlike prior works, we did not explicitly group characters into phonemes, allowing the rhythm encoder to automatically learn the most useful higher-order linguistic structure. An example of three FACS extracted from the LibriSpeech dataset is shown in Table 1 . The sequence features for all speech segments were generated offline prior to the training of the rhythm encoder. 3.2 Rhythm Encoder To leverage rhythmic information for speaker recognition, we trained an encoder model, shown in figure 2 , to predict speaker identity from the FACS. We elected to use a transformer-based architecture based on these models’ success in modeling patterns for sequence-based tasks. In particular, our model consisted

Read the complete text