HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech
Aurosweta Mahapatra, Ismail Rasim Ulgen · 2025
Read HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech by Aurosweta Mahapatra, Ismail Rasim Ulgen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21676.txt.
Introduction
Expressive synthetic speech fools anti spoofing systems that ignore prosody, the rhythm and melody of speech. Aurosweta Mahapatra and 2 co-authors build a prosody aware detector in a 2025 paper. Humans use pitch patterns and voiced structure to sense fakes instinctively. Their framework, HuLA, learns in two stages with multiple tasks. A self supervised backbone anchors the design.
In stage one, the backbone trains on real speech with pitch prediction and voiced classification as auxiliary tasks. That mirrors human perceptual learning of natural prosodic variation. In stage two, spoof detection and prosody tasks train jointly on real and synthetic data. Prosodic mismatches between natural and expressive synthetic speech become the detection cue. Experiments show gains for expressive and emotional fakes.
Catalog record
| CREATOR | Aurosweta Mahapatra, Ismail Rasim Ulgen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech Thanks: A. Mahapatra is with the Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA (e-mail: [email protected]). Thanks: Ismail Rasim Ulgen is with the Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA (e-mail: [email protected]). Thanks: B. Sisman is with the Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA (e-mail: [email protected]). Thanks: Manuscript received September 24, 2025. Aurosweta Mahapatra, Ismail Rasim Ulgen Abstract Current anti-spoofing systems remain vulnerable to expressive and emotional synthetic speech, since they rarely leverage prosody as a discriminative cue. Prosody is central to human expressiveness and emotion, and humans instinctively use prosodic cues such as F 0 F_{0} patterns and voiced/unvoiced structure to distinguish natural from synthetic speech. In this paper, we propose HuLA , a two-stage prosody-aware multi-task learning framework for spoof detection. In Stage 1, a self-supervised learning (SSL) backbone is trained on real speech with auxiliary tasks of F 0 F_{0} prediction and voiced/unvoiced classification, enhancing its ability to capture natural prosodic variation similar to human perceptual learning. In Stage 2, the model is jointly optimized for spoof detection and prosody tasks on both real and synthetic data, leveraging prosodic awareness to detect mismatches between natural and expressive synthetic speech. Experiments show that HuLA consistently outperforms strong baselines on challenging out-of-domain dataset, including expressive, emotional, and cross-lingual attacks. These results demonstrate that explicit prosodic supervision, combined with SSL embeddings, substantially improves robustness against advanced synthetic speech attacks. I Introduction Anti-spoofing aims to detect audio generated through replay attacks, speech synthesis, and voice conversion (VC) [ 1 ] . Recent progress in text-to-speech (TTS) [ 2 , 3 , 4 , 5 , 6 ] and VC systems [ 7 , 8 , 9 , 10 , 11 ] has amplified concerns about expressive synthetic speech, which can be misused to compromise biometric authentication or impersonate speakers for spreading misinformation [ 12 , 13 ] . These risks highlight the urgent need for robust and generalizable anti-spoofing systems. One of the goals of speech generation is to produce speech that is natural and indistinguishable from human speech. Expressiveness and emotion are the defining characteristics of human speech. TTS and VC systems often rely on prosody, particularly the fundamental frequency ( F_{0} ) to approximate these aspects [ 14 , 15 , 16 , 17 ] . However, accurately modeling F_{0} remains challenging [ 18 , 19 , 20 ] , and current synthesis systems do not fully capture the subtleties of human expressiveness. While this limitation is a weakness for synthesis, it represents a valuable opportunity for anti-spoofing: imperfect expressiveness can serve as a discriminative cue for detecting synthetic speech. Anti-spoofing research has been driven largely by community challenges such as ASVspoof [ 21 ] and the Audio Deep Synthesis Detection (ADD) Challenge [ 22 , 23 ] , which introduced benchmark datasets including ASVspoof 2019 [ 24 ] , 2021 [ 25 ] , and 2024 [ 26 ] . These resources have significantly advanced the field and now serve as standard evaluation benchmarks. Current approaches span three main categories: (i) end-to-end models that operate directly on raw waveforms (e.g., RawNet2 [ 27 ] , AASIST [ 28 ] ); (ii) methods based on handcrafted spectral features combined with supervised classifiers (e.g., LCNN [ 29 ] , ASSERT [ 30 ] , ResNet34 [ 31 ] ); and (iii) pretrained self-supervised learning (SSL) models such as wav2vec 2.0, HuBERT, and WavLM [ 32 , 33 , 34 , 35 ] . SSL-based approaches are now widely preferred due to their pretraining on large-scale data, ability to learn generalizable representations, and flexibility for fine-tuning in anti-spoofing tasks [ 36 , 37 , 38 ] . Despite these advances, current SSL-based models fail to address the imperfect replication of emotion and expressiveness in synthetic speech, an underexplored, discriminative cue. In this work, we focus on the prosodic dimension of expressiveness and explicitly incorporate prosodic information alongside SSL embeddings to enhance spoof detection. Prosody refers to suprasegmental features of speech [ 39 , 40 ] that shape perception beyond linguistic content, playing a central role in conveying expressiveness and emotion. Humans are inherently sensitive to prosodic cues due to lifelong exposure to natural emotional and expressive speech, and can often distinguish real from synthetic audio based on these cues [ 41 ] . Common prosodic attributes include fundamental frequency ( F_{0} ), jitter, shimmer, and speaking rate. Among these, F_{0} is most widely used in synthetic speech systems to improve naturalness [ 42 , 43 , 44 ] , supported by reliable pitch estimation algorithms such as YAAPT [ 45 ] , DIO [ 46 ] , and Harvest [ 47 ] . Despite its importance to human perception, prosody remains underexplored in anti-spoofing. Recent studies [ 48 , 49 , 50 ] have examined prosodic features in isolation [ 48 ] , in combination with speaker verification embeddings [ 49 ] , or through emotion recognition features [ 50 ] , achieving performance comparable to or exceeding baselines. These findings underscore the potential of prosody as a valuable yet underutilized cue for spoof detection, especially when explicitly integrated with SSL-based contextual embeddings. Inspired by this, we aim to make anti-spoofing models listen like a human by leveraging prosodic awareness to better distinguish between real and spoofed speech. To this end, we propose HuLA (Human-Like Listener for Anti-spoofing) , a two-stage prosody-aware multi-task learning framework. In Stage 1, a pretrained SSL model is fine-tuned on prosody-related tasks (e.g. F_{0} prediction and voiced/unvoiced (V-UV) classification) using only real speech. This stage is designed to improve the model’s understanding of natural prosodic variation. We extract F_{0} values at the frame level using a pitch extraction algorithm and use them as reference labels during training. To support the F_{0} prediction task, we incorporate V-UV frame classification. This helps the model learn voiced/unvoiced positioning within the F_{0} sequence which is an important aspect of prosodic structure. By learning these patterns in real speech, the model builds a foundation for detecting prosodic deviations in synthetic speech. In Stage 2, the model is jointly optimized for spoof detection and the same prosody tasks on both real and synthetic audio, exposing it to general differences while explicitly leveraging prosodic cues. HuLA consistently outperforms baselines on challenging out-of-domain datasets, including expressive/emotional speech (ASVspoof [ 25 , 26 ] , EmoFake [ 51 ] , mixed emotions [ 52 ] ) and cross-lingual data [ 22 , 53 ] . Our key contributions are as follows: • We identify imperfect modeling of expressiveness as both a vulnerability of speech synthesis and a discriminative opportunity for anti-spoofing. • We highlight the overlooked role of prosody, particularly F_{0} and voiced/unvoiced patterns, as a discriminative cue for detecting expressive synthetic speech, addressing a key gap in current anti-spoofing research. • We introduce HuLA, a two-stage multi-task framework that leverages prosody-awareness for improved spoof detection. • We demonstrate strong out-of-domain robustness, showing HuLA’s effectiveness against expressive, emotional, and cross-lingual spoofing attacks. The rest of this paper is organized as follows. Section II reviews related