Spectro-Temporal Modulation Representation Framework for Human-Imitated Speech Detection
Khalid Zaman, Masashi Unoki · 2026
Read Spectro-Temporal Modulation Representation Framework for Human-Imitated Speech Detection by Khalid Zaman, Masashi Unoki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23241.txt.
Introduction
Zaman and Unoki tackle human-imitated speech forgery, which fools people and detectors more easily than AI-made speech because it sounds natural. They propose a Spectro-Temporal Modulation frame built on hearing, using Gammatone and Gammachirp cochlear filterbanks to capture joint time and frequency shifts. A Segmental-STM variant studies short overlapping windows for fine-grained time detail. Tests show STM near human scores, while Segmental-STM beats them, backing hearing-inspired models for voice checks.
Catalog record
| CREATOR | Khalid Zaman, Masashi Unoki |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38644 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23241
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spectro-Temporal Modulation Representation Framework for Human-Imitated Speech Detection Khalid Zaman, Masashi Unoki Abstract Human-imitated speech poses a greater challenge than AI-generated speech for both human listeners and automatic detection systems. Unlike AI-generated speech, which often contains artifacts, over-smoothed spectra, or robotic cues, imitated speech is produced naturally by humans, thereby preserving a higher degree of naturalness that makes imitation-based speech forgery significantly more challenging to detect using conventional acoustic or cepstral features. To overcome this challenge, this study proposes an auditory perception-based Spectro-Temporal Modulation (STM) representation framework for human-imitated speech detection. The STM representations are derived from two cochlear filterbank models: the Gammatone Filterbank (GTFB), which simulates frequency selectivity and can be regarded as a first approximation of cochlear filtering, and the Gammachirp Filterbank (GCFB), which further models both frequency selectivity and level-dependent asymmetry. These STM representations jointly capture temporal and spectral fluctuations in speech signals, corresponding to changes over time in the spectrogram and variations along the frequency axis related to human auditory perception. We also introduce a Segmental-STM representation to analyze short-term modulation patterns across overlapping time windows, enabling high-resolution modeling of temporal speech variations. Experimental results show that STM representations are effective for human-imitated speech detection, achieving accuracy levels close to those of human listeners. In addition, Segmental-STM representations are more effective, surpassing human perceptual performance. The findings demonstrate that perceptually inspired spectro-temporal modeling is promising for detecting imitation-based speech attacks and improving voice authentication robustness. I Introduction Speech is widely used in modern technologies such as voice assistants, biometric systems, and online communication platforms. While these technologies have improved convenience and accessibility for users, they have also introduced new security concerns. In particular, speech signals have become increasingly vulnerable to manipulation and forgery, creating significant challenges for speaker verification, authentication, and privacy protection [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] . Among these emerging threats, AI-generated synthetic speech has attracted significant attention, as advanced techniques such as text-to-speech (TTS), voice conversion (VC), and deep learning–based synthesis can now produce highly realistic and natural-sounding speech. These synthesis approaches have rapidly advanced with the introduction of neural architectures such as Tacotron [ 8 , 9 ] , WaveNet [ 10 ] , and GAN-based [ 11 , 12 ] and diffusion-based vocoders [ 13 ] , which can generate highly natural and expressive speech by reproducing speaker-specific traits and emotional tones. Trained on large-scale speech corpora, these data-driven models learn to capture prosodic patterns, pitch contours, and vocal timbre with remarkable fidelity. To systematically evaluate the security implications of these technological advances, several benchmark corpora and detection challenges have been established, including ASVSpoof [ 14 , 15 , 16 , 17 , 18 , 19 ] , audio deepfake detection (ADD) [ 20 ] , [ 21 ] , Real or Fake [ 22 ] , Chinese Fake Audio Detection (CFAD) [ 23 ] , Wavfake [ 24 ] , and In-the-Wild (ITW) Wavfake [ 25 ] , along with related spoofing studies [ 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ] . These benchmarks have played a crucial role in identifying vulnerabilities in automatic speaker verification systems and in facilitating the development and comparison of anti-spoofing techniques under diverse attack scenarios and acoustic conditions. Despite the success of these datasets and challenges, most AI-generated speech still exhibits detectable artifacts. Such signals often sound relatively uniform or slightly robotic, with imperfections that arise from over-smoothed spectral envelopes, limited fine-grained prosodic variation, and temporal or phase distortions introduced by statistical averaging or model regularization. Consequently, although synthetic speech may sound natural at a coarse level, it typically lacks the subtle micro-level articulatory and modulation cues inherent to genuine human speech, making it comparatively easier for current detection systems to identify. In contrast, human-imitated speech presents a substantially more challenging detection problem. Human-imitated speech, produced by humans intentionally mimicking another speaker [ 34 ] , closely resembles genuine speech in its naturalness and expressiveness. It preserves pitch dynamics, rhythm, articulation patterns, and vocal timbre with high fidelity, often making it difficult for both human listeners and automatic systems to distinguish from genuine speech [ 5 ] , [ 34 ] . This challenge is further exacerbated by the scarcity of datasets specifically designed for imitation-based detection [ 5 ] . Unlike synthetic or voice-converted speech, genuine human imitation datasets are limited in size, lack standardized recording conditions, and are underrepresented in current benchmarks. As a result, existing spoofing detection systems, which are largely optimized for algorithmically generated speech, tend to perform poorly when confronted with naturally produced human-imitated speech. This performance gap highlights a critical limitation of current approaches and motivates the development of detection methods that better reflect perceptually relevant characteristics of speech signals. While several studies have attempted to address imitation-related detection, most prior work has focused on vocal or synthetic imitation rather than genuine human-imitated speech. For example, the vocal imitation dataset [ 35 ] and related signal-processing-based approaches [ 36 ] primarily investigate artificial or sound-level imitation. Other studies examine non-speech imitation, such as the Arabic cantillation corpus [ 37 ] , or musical imitation, as in the vocal percussion corpus [ 38 ] . Zetterholm [ 39 ] analyzed voice imitation using a small set of imitations of Swedish public figures, combining perceptual listening tests with acoustic analyses of features such as fundamental frequency, formants, articulation rate, and spectral characteristics of fricatives. However, the limited dataset size and lack of statistical modeling constrained the generalizability of the findings. Similarly, Hautamäki et al. [ 40 ] evaluated a Finnish mimicry corpus using both automatic speaker verification systems and human listeners, while Zaman et al. [ 41 ] examined human-imitated speech through subjective listening experiments and timbral feature analysis. However, both studies relied on limited acoustic representations, resulting in modest detection performance. Collectively, the existing methods often depend on descriptive analyses or low-level spectral features, which are insufficient to capture the subtle perceptual cues that distinguish genuine speech from high-quality human imitation. This observation underscores the need for more advanced and perceptually grounded acoustic representations. Temporal and spectral modulations within the modulation domain provide a representation of acoustic variations across time and frequency, as proposed by Flinker et al. [ 42 ] . Accordingly, recent studies in speech and audio analysis have shown that modeling spectro-temporal modulations (STM) provides a richer and more perceptually relevant representation of sound than conventional spectral or cepstral features. Wu et al. [ 43 ] introduced magnitude and phase modulation features for detecting synthetic speech, demonstrating that modulation-domain information improves spoofing detection performance. STM representations have also been applied to deepfake speech detection; for example, Cheng et al. [ 44 ] reported that STM features better simulate aspects of human auditory perception and enhance discrimination between genuine and fake speech by capturing subtle articulatory and prosodic cues. In addition, Li et al. [ 45 ] proposed an STM analysis framework with machine-specific filterbanks for anomaly detection. In this method, they first quantified machine-specific frequency importance using the Fisher ratio (F-ratio) and designed non-uniform filterbanks (NUFBs) to extract the Log Non-Uniform Spectrum (LNS), emphasizing discriminative frequency regions.