Multi-channel Replay Speech Detection using an Adaptive Learnable Beamformer

Michael Neri 1 {}^{\textbf{1}}, Member, IEEE, Tuomas Virtanen 1 {}^{\textbf{1}}, Fellow, IEEE · 2025

Read Multi-channel Replay Speech Detection using an Adaptive Learnable Beamformer by Michael Neri 1 {}^{\textbf{1}}, Member, IEEE, Tuomas Virtanen 1 {}^{\textbf{1}}, Fellow, IEEE (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13473.txt.

Introduction

Michael Neri and Tuomas Virtanen present a 2025 defense against replay attacks on voice systems, where a recording of a real user is played back to fool the lock. Their system listens with several microphones at once. A learnable beamformer, which aims the ear in space, is trained jointly with a neural classifier, so the spatial filter and the verdict tune each other.

On a standard multi-channel test set with four array shapes and four rooms, the system beat the prior state of the art. Gains were largest in hard acoustic settings. It also carried better to rooms it had never heard, a key trait for real deployments in homes and offices.

Catalog record

CREATORMichael Neri 1 {}^{\textbf{1}}, Member, IEEE, Tuomas Virtanen 1 {}^{\textbf{1}}, Fellow, IEEE
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13473

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-channel Replay Speech Detection using an Adaptive Learnable Beamformer Michael Neri 1 {}^{\textbf{1}}, Member, IEEE, Tuomas Virtanen 1 {}^{\textbf{1}}, Fellow, IEEE Abstract Replay attacks belong to the class of severe threats against voice-controlled systems, exploiting the easy accessibility of speech signals by recorded and replayed speech to grant unauthorized access to sensitive data. In this work, we propose a multi-channel neural network architecture called M-ALRAD for the detection of replay attacks based on spatial audio features. This approach integrates a learnable adaptive beamformer with a convolutional recurrent neural network, allowing for joint optimization of spatial filtering and classification. Experiments have been carried out on the ReMASC dataset, which is a state-of-the-art multi-channel replay speech detection dataset encompassing four microphones with diverse array configurations and four environments. Results on the ReMASC dataset show the superiority of the approach compared to the state-of-the-art and yield substantial improvements for challenging acoustic environments. In addition, we demonstrate that our approach is able to better generalize to unseen environments with respect to prior studies. AE autoencoder ASV automatic speaker verification BCE binary cross-entropy CM countermeasure CNN convolutional neural network COTS commercial-off-the-shelf CQCC constant-Q cepstral coefficient CRNN convolutional recurrent neural network DNN deep neural network DF deepfake EER equal error rate ELU exponential linear unit FCN fully convolutional network GMM Gaussian mixture model GRU gated recurrent unit IoT Internet of things LA logical access LSTM long short-term memory MVDR minimum variance distortionless response MWF multichannel Wiener filtering ReMASC Realistic Replay Attack Microphone Array Speech RNN recurrent neural network ROC receiving operating characteristic TTS text-to-speech PA physical access PFA probability of false alarm SOTA state-of-the-art STFT short-time Fourier transform VA voice assistant VC voice conversion {IEEEkeywords} Replay attack, Physical Access, Beamforming, Spatial Audio, Voice anti-spoofing. † † corresponding: Corresponding author: Michael Neri (email: [email protected]). 1 Introduction In recent years, the use of VA has been increasing for human-machine interaction, employing voice as a biometric trait [ 1 ] . With VA , it is possible to control smart devices in our homes using IoT ( IoT ) networks and transmit sensitive information over the Internet. For this reason, malicious users started to attack audio-based data to grant unauthorized access to personal data and devices. Generally, there are several types of attacks that a malicious user can exploit toward an audio recording [ 2 ] . It is possible to perform a LA ( LA ) attack, i.e., applying TTS ( TTS ) and/or VC ( VC ) algorithms to modify the speech content or imitate the user. Lately, the DF ( DF ) attack showed how compression and quantization operations to audio recordings can hide LA artifacts. Figure 1: The problem of detecting the replay speech attack. In this scenario, a genuine speech from a talker can be either directly transmitted to a biometric system or intercepted by a spoofing microphone, recorded, and replayed through a speaker. The replayed speech, captured by the system, may be misclassified as genuine, granting unauthorized access to private data. A countermeasure and automatic speaker verification module are typically used to distinguish between genuine speech and spoofed (replayed) speech. Differently, with PA ( PA ) attacks the objective is to counterfeit the ASV ( ASV ) system at the microphone level. In this category, there are impersonation attacks [ 1 ] and replay attacks [ 3 ] . The former involves an attacker who is able to mimic the user speech of the target to grant access through the ASV system. The latter, instead, consists of a fraudulent recording of the genuine speaker using a spoofing microphone, which is later replayed to the ASV microphone. An example of a replay attack is shown in Figure 1 . The research community has mainly focused its attention on LA attacks by developing CM ( CM ) models that are responsible for detecting synthethic generation artifacts. In fact, several datasets have been published with the aim to tackle these attacks, e.g., the ASVSpoof series [ 4 , 2 ] . However, voice is different to other biometric markers as it is always publicly exposed, easing the collection of spoofed recordings by attackers from the target speaker [ 5 ] . This characteristic allows malicious users to easily perform PA even without any professional audio expertise. In fact, replay attacks can be performed by using COTS ( COTS ) devices like smartphone and loudspeakers with high success rates. In the literature, several works have tried to mitigate this type of attack by providing single-channel speech datasets, such as RedDots [ 6 ] , ASVSpoof2017 PA [ 7 ] , ASVSpoof2019 [ 8 ] PA , and ASVSpoof2021 PA [ 2 ] . Building upon these datasets, researchers have been deploying various methods to address single-channel replay attack detection, such as capsule networks [ 9 ] , CNN with hand-crafted audio features combined with time-frequency representations [ 10 , 11 , 12 ] , and teacher-student architectures to reduce the computational complexity for real-time detection [ 13 ] . However, for speech enhancement and separation tasks, microphone arrays are usually employed in ASV systems to exploit spatial information and improve audio quality [ 14 ] . Moreover, multi-channel data can be beneficial for detecting the replay detection for several reasons: (i) multi-channel recordings encompass audio spatial cues that can help the detection [ 15 , 3 ] , and (ii) this spatial information cannot be easily counterfeited by an attacker, differently to single-channel data where temporal and frequency cues can be manipulated to fool an ASV system [ 16 ] . Unfortunately, single-channel replay detectors do not exploit spatial information from multi-channel recordings. In this regard, single-channel replay detectors demonstrated poor generalization capabilities across different environments and devices on multi-channel data [ 3 ] . Moreover, there is a lack of datasets and models that are capable to collect and detect replay attacks based on multi-channel recordings. ReMASC ( ReMASC ) is the only publicly available dataset that encompasses real multi-channel audio samples, including different microphone arrays, playback devices, and environments. The baseline of this dataset was CQCC-GMM [ 15 ] , which is a GMM ( GMM ) classifier trained on CQCC . The authors of the database also proposed a CRNN ( CRNN ) which acts as both a learnable and adaptive time-domain beamformer with a CRNN replay attack classifier [ 3 ] . Recent works in SOTA ( SOTA ) focused on hand-crafted single-channel features with simple classifiers (such as CNN and GMM ) after a beamforming phase (TECC [ 17 ] , CTECC [ 18 ] , and ETECC [ 19 ] ). However, all these methods suffer from generalization capabilities, i.e., changing the acoustic properties on the test set yields random guess predictions. This problem also occurs with single-channel utterances [ 11 ] . In addition, performance for known environments is still not enough for production stage. To tackle the aforementioned problems, the contributions of this work are: • The definition of a novel M ulti-channel R eplay A ttack D etector that exploits a L earnable time-frequency A daptive beamformer to classify between genuine and spoofed audio recordings, namely M-ALRAD . It includes a time-frequency beamformer and a CRNN classifier. •

Read the complete text