Addressing Gradient Misalignment in Data-Augmented Training for Robust Speech Deepfake Detection
Duc-Tuan Truong, Tianchi Liu, Junjie Li, Ruijie Tao, Kong Aik Lee, Eng Siong Chng · 2025
Read Addressing Gradient Misalignment in Data-Augmented Training for Robust Speech Deepfake Detection by Duc-Tuan Truong, Tianchi Liu, Junjie Li, Ruijie Tao, Kong Aik Lee, Eng Siong Chng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20682.txt.
Introduction
Data augmentation helps speech deepfake detectors handle new voices and attacks, but it can also disturb training. Duc-Tuan Truong and five co-authors study that side effect in a 2025 paper accepted for ICASSP 2026. The problem is gradient misalignment between original and altered speech.
When the two versions push the model in conflicting directions, training can settle in a worse spot. The authors build a dual-path framework that runs each utterance in original and augmented form side by side. It then compares the two gradient directions and aligns them to cut conflicts. Experiments reported in the paper show more robust detection. The work matters as fake speech grows harder to catch.
Catalog record
| CREATOR | Duc-Tuan Truong, Tianchi Liu, Junjie Li, Ruijie Tao, Kong Aik Lee, Eng Siong Chng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20682
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Addressing Gradient Misalignment in Data-Augmented Training for Robust Speech Deepfake Detection Duc-Tuan Truong, Tianchi Liu, Junjie Li, Ruijie Tao, Kong Aik Lee, Eng Siong Chng Abstract In speech deepfake detection (SDD), data augmentation (DA) is commonly used to improve model generalization across varied speech conditions and spoofing attacks. However, during training, the backpropagated gradients from original and augmented inputs may misalign, which can result in conflicting parameter updates. These conflicts could hinder convergence and push the model toward suboptimal solutions, thereby reducing the benefits of DA. To investigate and address this issue, we design a dual-path data-augmented (DPDA) training framework with gradient alignment for SDD. In our framework, each training utterance is processed through two input paths: one using the original speech and the other with its augmented version. This design allows us to compare and align their backpropagated gradient directions to reduce optimization conflicts. Our analysis shows that approximately 25% of training iterations exhibit gradient conflicts between the original inputs and their augmented counterparts when using RawBoost augmentation. By resolving these conflicts with gradient alignment, our method accelerates convergence by reducing the number of training epochs and achieves up to an 18.69% relative reduction in Equal Error Rate on the In-the-Wild dataset compared to the baseline 1 1 1 Code and models are available at github.com/ductuantruong/dpda_ga . † † address: 1 Nanyang Technological University, Singapore 2 National University of Singapore, Singapore 3 The Hong Kong Polytechnic University, Hong Kong 1 Introduction Speech deepfake detection (SDD) models perform well under seen conditions but struggle to generalize to unseen scenarios. They often suffer from significant accuracy drops under new attack types, acoustic conditions, or noise that can hide synthetic speech artifacts [ 1 , 2 ] . To address this challenge, data augmentation (DA) is a straightforward strategy to increase training variability for SDD. Common DA techniques in speech processing include room impulse response (RIR) [ 3 , 4 ] , environmental noise addition [ 5 ] , and masking [ 6 , 7 ] . Based on that, RawBoost [ 8 ] is designed for the SDD task by applying signal processing to distort raw waveforms, which enhances robustness to acoustic variations. Recent studies have explored different SDD augmentation strategies. For example, [ 9 ] proposes an adversarial attack-inspired method that produces pseudo-fake samples near decision boundaries. CpAug [ 10 ] applies a copy–paste strategy to enhance intra-class diversity. Additionally, [ 11 ] explores embedding-level augmentation to diversify the feature space and promote more generalizable representations. While DA improves the robustness of SDD, prior work [ 4 ] has shown that DSD models trained with DA still suffer performance degradation when evaluated under new acoustic conditions within the same test set [ 4 ] . This suggests that DA alone is insufficient and better results may be achieved by helping the model focus distinguishing bona fide from spoofed audio rather than overfitting to acoustic variations introduced by augmentation. We hypothesize that the sensitivity of SDD models to DA perturbations leads to different loss surface between original and augmented inputs. As a result, gradients backpropagated from these two input types may point in different directions, causing gradient conflicts during joint updates. These conflicts can slow down convergence, and push optimization toward suboptimal solutions. Similar issues have been observed in multi-task learning [ 12 ] , where gradients from different tasks can be conflicting and harm training [ 12 ] . Gradient alignment methods were originally introduced in multi-task training to reduce such conflicts and stabilize optimization [ 13 , 14 , 15 ] . These methods were later extended to domain generalization to enforce gradient consistency across domains [ 16 ] . Motivated by these observations, we propose a dual-path data-augmented (DPDA) training framework with gradient alignment for speech deepfake detection (SDD). In this framework, the model is trained using both original and augmented version of each training utterance, and a gradient alignment method is applied when their backpropagated gradients conflict. Although gradient alignment methods has been studied in other contexts such as multi-task learning and domain generalization, our approach is distinct in that it aligns gradients between two inputs derived from the same utterance but modified through DA. This alignment encourages consistent parameter updates and helps the model focus on spoof-related cues rather than overfitting to augmentation artifacts. To the best of our knowledge, this is the first work to investigate gradient conflicts during data-augmented training in the context of SDD. Our study provides the following key findings: (i) gradient conflicts between the original and augmented inputs are linked to differences in their loss landscapes; (ii) such conflicts occur frequently during DPDA training; and (iii) resolving them consistently improves performance across multiple SDD architectures, augmentation strategies, and benchmark datasets over both our DPDA baseline and conventional single-path data-augmented training. Finally, our approach is architecture-agnostic and can be easily integrated with existing models and signal-level data augmentation techniques. Figure 1: Overview of the dual-path data-augmented (DPDA) training framework with gradient alignment for SDD. The left panel illustrates the overall training framework, wherein both original and augmented inputs are processed in parallel. The right panel compares three gradient alignment methods, including PCGrad [ 13 ] , GradVac [ 14 ] , and CAGrad [ 15 ] , which are used to resolve gradient conflicts. 2 Methodology Figure 2: Training loss and backpropagated gradient norm of orignal x and augmented \tilde{x} inputs during the DPDA training of XLSR-Conformer-TCM model. 2.1 The proposed training framework Figure 1 illustrates our proposed DAPA training framework with gradient alignment. Let x denote a original input speech sample and the augmented counterpart \tilde{x} generated through an input-level data augmentation method. Both inputs are formed as a mini-batch and passed through a shared SDD model f(\theta) with parameters \theta to compute the loss \mathcal{L} . In conventional training, the gradients from both inputs are aggregated directly as \mathbf{g}=\nabla_{\theta}\mathcal{L}([x,\tilde{x}]) . However, although x and \tilde{x} come from the same utterance, their gradients may differ, and direct aggregation can introduce conflicts that hinder optimization. To investigate this, we compare the training losses \mathcal{L}(x) and \mathcal{L}(\tilde{x}) alongside the gradient magnitudes computed from each input path. As shown in Figure 2 , there is a clear pattern that the training loss of the augmented input path is higher than the original one, hence lead to a larger gradient norms of augmented input compared to the orignal one. This imbalance can cause the model updates to be dominated by the augmented input. As a result, this dominance may bias the model toward learning augmentation-specific patterns, rather than focusing on spoof-related cues. The issue can become worse when the gradients from x and \tilde{x} point in different directions, which can introduce conflicts during optimization. To further investigate the source of gradient conflicts between original and augmented inputs, we visualize the local loss surfaces of the model for both inputs at an intermediate point during training. Following the method introduced by [ 17 ] , we approximate the loss landscape around the model parameters by perturbing them along two randomly sampled, orthogonal directions in parameter space. The directions are normalized and scaled, and the loss is evaluated on a grid around the current parameter point, forming