SOURCE TRACING OF SYNTHETIC SPEECH SYSTEMS THROUGH PARALINGUISTIC PRE-TRAINED REPRESENTATIONS † Contributed equally as first authors

Girish 1 3 † , Mohd Mujtaba Akhtar 2 † , Orchid Chetia Phukan 3 † , Drishti Singh 3 † Swarup Ranjan Behera 4 , Pailla Balakrishna Reddy 5 , Arun Balaji Buduru 1 , Rajesh Sharma 6 7 · 2025

Read SOURCE TRACING OF SYNTHETIC SPEECH SYSTEMS THROUGH PARALINGUISTIC PRE-TRAINED REPRESENTATIONS † Contributed equally as first authors by Girish 1 3 † , Mohd Mujtaba Akhtar 2 † , Orchid Chetia Phukan 3 † , Drishti Singh 3 † Swarup Ranjan Behera 4 , Pailla Balakrishna Reddy 5 , Arun Balaji Buduru 1 , Rajesh Sharma 6 7 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01157.txt.

Introduction

Girish and co-authors published this speech technology paper in 2025. It tackles source tracing for synthetic speech: given a fake voice clip, can a system tell which generator made it? Each generator leaves its own fingerprints in pitch, tone, rhythm, and intonation. Earlier work drew features from general speech models. The team instead tests models pre-trained on paralinguistic traits, the expressive layer of speech, which they expect to carry stronger source cues.

Their comparison across many pre-trained models backed the guess: paralinguistic features traced sources best. They then built TRIO, a framework that fuses two such models with a gating mechanism and alignment training. TRIO beat single models and older fusion methods, setting a new best score for the task. The work matters for policing synthetic media, where knowing the tool behind a fake is a first step.

Catalog record

CREATORGirish 1 3 † , Mohd Mujtaba Akhtar 2 † , Orchid Chetia Phukan 3 † , Drishti Singh 3 † Swarup Ranjan Behera 4 , Pailla Balakrishna Reddy 5 , Arun Balaji Buduru 1 , Rajesh Sharma 6 7
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01157

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SOURCE TRACING OF SYNTHETIC SPEECH SYSTEMS THROUGH PARALINGUISTIC PRE-TRAINED REPRESENTATIONS † Contributed equally as first authors Girish 1 3 † , Mohd Mujtaba Akhtar 2 † , Orchid Chetia Phukan 3 † , Drishti Singh 3 † Swarup Ranjan Behera 4 , Pailla Balakrishna Reddy 5 , Arun Balaji Buduru 1 , Rajesh Sharma 6 7 Abstract In this work, we focus on source tracing of synthetic speech generation systems (STSGS). Each source embeds distinctive paralinguistic features—such as pitch, tone, rhythm, and intonation—into their synthesized speech, reflecting the underlying design of the generation model. While previous research has explored representations from speech pre-trained models (SPTMs), the use of representations from SPTM pre-trained for paralinguistic speech processing, which excel in paralinguistic tasks like synthetic speech detection, speech emotion recognition has not been investigated for STSGS. We hypothesize that representations from paralinguistic SPTM will be more effective due to its ability to capture source-specific paralinguistic cues attributing to its paralinguistic pre-training. Our comparative study of representations from various SOTA SPTMs, including paralinguistic, monolingual, multilingual, and speaker recognition, validates this hypothesis. Furthermore, we explore fusion of representations and propose TRIO , a novel framework that fuses SPTMs using a gated mechanism for adaptive weighting, followed by canonical correlation loss for inter-representation alignment and self-attention for feature refinement. By fusing TRILLsson (Paralinguistic SPTM) and x-vector (Speaker recognition SPTM), TRIO outperforms individual SPTMs, baseline fusion methods, and sets new SOTA for STSGS in comparison to previous works. I INTRODUCTION Advancements in audio manipulation technology have blurred the line between real and synthetic speech. Modern text-to-speech (TTS) and voice conversion (VC) systems can produce highly realistic voices, enabling malicious actors to manipulate speech with remarkable accuracy. This growing challenge highlights the urgent need for reliable methods to detect and attribute synthetic speech. This advancement poses significant risks, as malicious entities can exploit synthetic speech for impersonation, fraud, and misinformation. As such the urgent need for robust synthetic speech detection (SSD) solutions becomes undeniable to safeguard trust in digital communication. As a remedy there has been sufficient research into SDD [ 1 , 2 , 3 ] . Also, the use of representations from speech pre-trained models (SPTMs) such as Wav2vec2, WavLM, Whisper have captured recent attention within the community as these SPTMs provide performance benefit [ 4 , 5 , 6 ] . These SPTMs are either fine-tuned or used as feature extractors for extracting representations. Despite much advancement in synthetic speech detection, most of the previous research has mostly focused on distinguishing real and synthetically generated speech i.e. binary classification, but it is not sufficient to predict and mitigate misuse and improve forensic analysis. To further enhance synthetic speech detection from forensic analysis, it is important to understand the exact tool used to generate the speech and this task is known as Source Tracing of Synthetic Speech Generation Systems (STSGS). It has recently captured attention within the community and plays a crucial role in improving the explainability of detection systems, enforcing accountability, and developing targeted countermeasures against malicious deepfake applications [ 7 , 8 , 9 ] . Each source (TTS, VC) imprint distinctive paralinguistic features—including pitch, tone, rhythm, and intonation—onto their synthesized speech, mirroring the underlying design principles and processing mechanisms of the respective generation models. As such previous research on STSGS have investigated representations from various state-of-the-art (SOTA) SPTMs [ 10 , 11 , 12 , 13 ] for understanding their capability for capturing such source-specific paralinguistic cues. However, they haven’t investigated the usage of representations from SPTM pre-trained for paralinguistic speech processing such as TRILLsson [ 14 ] which have shown SOTA behavior for different paralinguistic tasks including tasks such as synthetic speech detection and speech emotion recognition. In this work, we solve this research gap and explore representations from paralinguistic SPTM for STSGS. We hypothesize that paralinguistic SPTM representations will be the most effective for STSGS, as their specialized paralinguistic pre-training enables them to capture paralinguistic cues unique to each source more effectively than other SPTM representations . To test this hypothesis, we conduct a comprehensive comparative study of various SOTA SPTMs, including paralinguistic, monolingual, multilingual, and speaker recognition. Our findings validate our hypothesis. Additionally, inspired by prior research demonstrating performance gains through SPTMs representations fusion in related areas such as synthetic speech detection [ 15 ] and speech emotion recognition [ 16 ] , we also explore this direction for STSGS. Phukan et al. [ 13 ] have made the initial exploration for fusion of SPTMs representation for STSGS, however, they have considered only a handful of SPTMs representations, here, in our study, we consider a wide range of SOTA SPTMs representations and also the inclusion of paralinguistic SPTM representations that has been missing and a major drawback in their study. To this end, we introduce TRIO (Ga T ed Canonical Cor R elat IO n Attention Network), a novel framework for fusing SPTMs. TRIO employs a gated mechanism for adaptive weighting of representations, incorporates canonical correlation loss for better alignment between the representations, and utilizes self-attention for enhanced feature refinement. By fusing TRILLsson (a paralinguistic SPTM) with x-vector (a speaker recognition SPTM), TRIO achieves superior performance, outperforming individual SPTMs, baseline fusion techniques, and setting a new SOTA benchmark for STSGS in comparison to previous works. In summary, the key contributions of this work are as follows: • We carry out a comprehensive comparative analysis of various SOTA SPTMs representations to understand the capability of paralinguistic SPTM representations for STSGS. We show that representations from TRILLsson achieves the topmost performance amongst all other SPTMs representations. • We introduce a novel framework, TRIO for effective fusion of SPTMs representations. TRIO uses a gated mechanism for adaptive representation weighting, applies canonical correlation loss for improved alignment, and employs self-attention for refined feature enhancement. By fusing TRILLsson and x-vector, TRIO surpasses individual SPTMs and baseline fusion methods, setting a new SOTA benchmark for STSGS compared to prior works. To make our work more accessible and reproducible, we’ve shared the code and models at 1 1 1 https://github.com/Helix-IIIT-Delhi/TRIO-Source_Tracing . II Speech Pre-trained Representations In this section, we present a brief overview of the SPTMs used in our study. Wav2Vec2 2 2 2 https://huggingface.co/facebook/wav2vec2-base [ 17 ] , WavLM 3 3 3 https://huggingface.co/microsoft/wavlm-base [ 18 ] , and Unispeech-SAT 4 4 4 https://huggingface.co/microsoft/unispeech-sat-base [ 19 ] are monolingual SPTMs and we consider their base versions pre-trained on LibriSpeech (960 hours of English). Wav2Vec2 was trained to solve a contrastive learning objective, WavLM was pre-trained for solving masked speech modeling and speech denoising simultaneously while Unispeech-SAT was trained in a multi-task speaker-aware format. Both WavLM and Unispeech-SAT have reported SOTA performance in SUPERB. We consider XLS-R 5 5 5 https://huggingface.co/facebook/wav2vec2-xls-r-300m [ 20 ] , Whisper 6 6 6 https://huggingface.co/openai/whisper-base [ 21 ] , and MMS 7 7 7 https://huggingface.co/facebook/mms-1b [ 22 ] for multilingual SPTMs. We consider their 300M, 74M, and 1B parameters version for XLS-R, Whisper, and MMS respectively. XLS-R, Whisper, MMS were pre-trained on 128, 96, and over 1400 languages respectively. XLS-R and MMS follows Wav2vec2 architecture and pre-trained in a contrastive learning approach while Whisper is a vanilla transformer encoder-decoder architecture and trained in a multi-task manner. We also consider speaker recognition SPTMs

Read the complete text