Probing-Guided Layer Selection from Self-Supervised Speech Models for Generalizable Audio Deepfake Detection

Marjan Beheshti, Majid Rostami, Bo Chen · 2026

Read Probing-Guided Layer Selection from Self-Supervised Speech Models for Generalizable Audio Deepfake Detection by Marjan Beheshti, Majid Rostami, Bo Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30791.txt.

Introduction

Marjan Beheshti, Majid Rostami and Bo Chen propose probing a speech model's layers before training a deepfake detector. Detectors often fail on unseen attacks because their features tie to specific fakes or recording setups. Self-supervised models hold features at many depths, but builders either pick one layer or fuse all of them, learning which mattered only after training.

Their two-stage method runs gradient-boosted probes on each transformer layer, ranking layers by cross-domain discrimination. A classifier then fuses the chosen layers, with the backbone frozen. Testing across three backbones revealed that layers cluster in depth zones. Swapping layers inside a zone changed little, while breaking the zone hurt, up to fivefold. On one backbone, four probed layers reached an equal error rate near five percent on in-the-wild data. They used one point three million trainable parameters, beating the best earlier result by twenty-eight percent.

Catalog record

CREATORMarjan Beheshti, Majid Rostami, Bo Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30791

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Probing-Guided Layer Selection from Self-Supervised Speech Models for Generalizable Audio Deepfake Detection Marjan Beheshti, Majid Rostami, Bo Chen Abstract Audio deepfake detection systems often fail to generalize across domains because they rely on features tied to specific attacks or recording conditions. Self-supervised speech models offer rich multi-layer representations, yet existing approaches either use a single layer or fuse all layers indiscriminately, and only reveal layer importance after training. We propose a model-agnostic, two-stage methodology that identifies informative depth zones before any task-specific model is trained. In the first stage, lightweight XGBoost probes evaluate each transformer layer’s cross-domain discriminative power, producing a layer ranking. In the second stage, a compact neural classifier fuses only the selected layers through per-layer attention pooling and a shared bottleneck projection, while the backbone remains frozen. Applied across three backbones, the probing reveals two key findings. First, informative layers cluster in depth zones rather than at uniquely optimal positions: within-zone substitutions fall within multi-seed noise, while zone violations degrade performance by up to 5 × \times . Second, the probing produces backbone-specific selections rather than a fixed layer recipe. On XLS-R-300M, four probing-selected layers with 1.34M trainable parameters achieve 4.94 ± \pm 0.32% equal error rate on In-The-Wild and 5.07% cross-domain average over four shared datasets, a 28% relative improvement over the best prior frozen-backbone result (Xiao and Vu, 2025) using all 25 layers with identical training data. 1 Introduction Modern text-to-speech and voice conversion systems can clone a speaker’s voice from as little as three seconds of audio [ 2 ] , producing fakes that human listeners cannot reliably distinguish from real speech [ 3 ] . The threat is already real: in early 2024, an employee at the engineering firm Arup authorized $25 million in wire transfers during a video call where every other participant was a deepfake [ 4 ] , and industry estimates place deepfake-enabled fraud losses in the hundreds of millions of dollars per year [ 5 , 6 ] . Beyond fraud, voice cloning enables threats ranging from CEO impersonation to bypassing biometric authentication [ 7 , 8 ] . Since humans cannot reliably detect these fakes, automated detection is essential. Crucially, such detectors must generalize across unseen attacks and recording conditions. We define a domain as the combination of attack algorithms, recording conditions, and speaker populations that characterize a dataset. In-domain evaluation tests on data drawn from the same domain as training, while cross-domain evaluation tests on data from entirely different domains—unseen attack types, different acoustic environments, or different speaker populations. Most existing deepfake detectors focus on in-domain accuracy, but maintaining detection performance across such domain shifts is still an open problem. Müller et al. [ 9 ] constructed the In-The-Wild dataset from real-world deepfake attacks found online, but their detectors that achieved near-perfect accuracy on ASVspoof benchmarks degraded to error rates greater than 30% on unseen samples. The degradation appears across architectures. AASIST [ 10 ] , a spectro-temporal graph attention network, achieves 0.83% equal error rate (EER) on ASVspoof 2019 LA Eval but increases to 34.8% on In-The-Wild [ 3 ] . RawNet2 [ 11 ] follows a similar degradation trajectory. These models learn to detect specific vocoder artifacts present in their training data rather than acquire a transferable notion of what makes speech synthetic. This cross-domain collapse stems from the feature extractor. Conventional front-ends, whether handcrafted or learned end-to-end, are built on a narrow acoustic distribution and overfit to the specific vocoder signatures present in the training data. Self-supervised speech models offer a fundamentally different starting point. Pre-trained on hundreds of thousands of hours of diverse, multilingual speech through task-agnostic objectives, models in the wav2vec 2.0 family 1 1 1 The wav2vec 2.0 family includes Base (12 layers, d{=}768 ) and Large (24 layers, d{=}1024 ) configurations. XLS-R additionally offers 1B (48 layers) and 2B (48 layers) variants. WavLM follows the same Base/Large split. In this work, we use exclusively the Large configurations: XLS-R-300M, WavLM Large, and XLSR-53, all with 24 transformer layers and d{=}1024 . [ 12 ] —including XLSR-53, XLS-R [ 15 ] , and WavLM [ 16 ] —learn representations that capture general properties of speech rather than dataset-specific artifacts. Crucially, these representations are not monolithic. Pasad et al. [ 17 , 18 ] demonstrated that the 24 transformer layers encode qualitatively different information along a consistent gradient: early layers capture acoustic and spectral properties, middle layers encode phonetic and prosodic structure, and late layers represent higher-level linguistic content. This hierarchy matters for deepfake detection because synthesis artifacts are themselves heterogeneous. Vocoder-based attacks leave spectral traces best captured by early-to-mid layers [ 19 ] , while TTS systems introduce statistical irregularities in prosody and linguistic structure that only late layers represent [ 20 ] . Consequently, the 24 layers vary widely in their utility for detection, raising the fundamental questions that this paper aims to address: 1) which layers carry the most discriminative information for cross-domain detection, and 2) how should they be combined? Not all layers contribute equally: El Kheir et al. [ 22 ] found that truncating to the first 10–12 layers matches or exceeds full-model performance, and Serrano et al. [ 21 ] showed that all-layer fusion degrades out-of-domain EER compared to the best individual layer. Yet existing work has explored three strategies for layer selection, none of which exploits this observation directly. The first strategy selects a single layer, typically the last hidden state or one chosen by validation [ 23 , 24 ] . This is practical and has proven effective on benchmarks where training and test distributions are similar, but restricts the detector to a single level of abstraction, discarding the acoustic, phonetic, or linguistic representations encoded at other depths. The second strategy fuses all layers. Martín-Doñas et al. [ 25 ] learn a weighted combination of all 25 hidden states, and Pan et al. [ 26 ] apply attention-based merging across all layers. Both learn implicit layer weights but include all representations, even those that Xiao and Vu [ 1 ] showed individually produce EER above 30% on In-The-Wild. The third strategy, also from Xiao and Vu [ 1 ] , trains 25 separate one-class softmax classifiers and sums their output scores at inference. Their post-hoc analysis confirms that layers 4–8 and 18–21 carry most of the discriminative signal, but the architecture still uses all 25 layer classifiers with equal score contribution, rather than removing the layers their own analysis identifies as uninformative. Across all three strategies, layer selection is absent or performed only after the full model is trained, providing no diagnostic information on which depth zones carry cross-domain discriminative information. The core difficulty is that layer importance is not uniform across domains: as we show in Section 5.1 , no single layer dominates all evaluation datasets, and in-domain validation alone cannot reliably identify which layers will generalize. In this work, we propose a two-stage approach that addresses this gap through principled layer selection before classifier training. Specifically in the first stage, we train independent lightweight classifiers on the frozen hidden states of each self-supervised learning (SSL) transformer layer and evaluate them across multiple cross-domain datasets, producing a layer ranking grounded in empirical discriminative power.

Read the complete text