Listen, Analyze, and Adapt to Learn New Attacks: An Exemplar-Free Class Incremental Learning Method for Audio Deepfake Source Tracing

Yang, Rohan Kumar · 2025

Read Listen, Analyze, and Adapt to Learn New Attacks: An Exemplar-Free Class Incremental Learning Method for Audio Deepfake Source Tracing by Yang, Rohan Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14601.txt.

Introduction

Yang Xiao and Rohan Kumar Das present a 2025 computing paper on tracing fake speech back to its source. Tools that make fake voices keep changing, so a detector must learn new attacks without forgetting old ones. That loss of old skill is known as catastrophic forgetting, and it is the core problem here.

Their method, called AnaST, learns new attack classes in a fresh way. The part of the model that reads speech stays fixed, while the part that sorts classes is updated with a closed-form answer in a single pass. The design guards privacy and uses little memory, so it suits online training. Tests reported in the paper show the method beats the baselines it was compared with.

Catalog record

CREATORYang, Rohan Kumar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14601

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Listen, Analyze, and Adapt to Learn New Attacks: An Exemplar-Free Class Incremental Learning Method for Audio Deepfake Source Tracing Yang, Rohan Kumar Abstract As deepfake speech becomes common and hard to detect, it is vital to trace its source. Recent work on audio deepfake source tracing (ST) aims to find the origins of synthetic or manipulated speech. However, ST models must adapt to learn new deepfake attacks while retaining knowledge of the previous ones. A major challenge is catastrophic forgetting, where models lose the ability to recognize previously learned attacks. Some continual learning methods help with deepfake detection, but multi-class tasks such as ST introduce additional challenges as the number of classes grows. To address this, we propose an analytic class incremental learning method called AnaST. When new attacks appear, the feature extractor remains fixed, and the classifier is updated with a closed-form analytical solution in one epoch. This approach ensures data privacy, optimizes memory usage, and is suitable for online training. The experiments carried out in this work show that our method outperforms the baselines. Xiao Das The University of MelbourneAustralia Fortemedia SingaporeSingapore keywords continual learning, class incremental learning, source tracing, audio deepfake detection † † email: [email protected], [email protected] 1 Introduction In recent years, deepfake generation and detection have drawn much attention [ 1 , 2 , 3 ] . Many global competitions, such as ASVspoof and the ADD challenge series [ 4 , 5 , 6 , 7 , 8 ] , encourage research on developing advanced and effective deepfake countermeasures. As binary classification (real/fake) has improved over the past few years, studies now focus on tracing the source of audio deepfake. This source tracing (ST) task [ 9 , 10 , 11 , 12 , 13 ] aims to find the system that produced a given speech sample. For example, law enforcement needs to find the source of legal cases, and developers want to protect their intellectual property. Thus, recognizing the audio deepfake algorithm is very important. Several ST methods have been proposed recently [ 9 , 10 , 11 ] , where various model designs and processing techniques are used that work well in controlled test sets. However, speech generation technology has witnessed continuous progress and nowadays deepfake speech sounds very close to real ones. This makes it difficult for current ST methods to handle unseen attacks. To address this, researchers have developed two main approaches. The first one uses data augmentation and multi-feature fusion to extract strong audio features, helping the model to manage many types of attacks [ 12 , 13 , 14 ] . The second one uses continual learning (CL), where the model learns from both previous and new data over time. However, the key problem in CL is catastrophic forgetting , which makes the model lose previously learned knowledge [ 15 ] . Specifically, the ST model must remember previously learned audio deepfakes attacks, while learning unseen new ones. To overcome catastrophic forgetting, CL [ 16 ] methods aim to learn new information while retaining the previous knowledge. This approach has been used in speech processing, audio classification, and other fields [ 17 , 18 , 19 ] . For instance, the authors in [ 20 ] propose “detecting fake without forgetting” (DFWF) for audio deepfake detection. This method uses regularization and a knowledge distillation loss to improve the model. However, audio deepfake detection is a binary classification task, where the number of classes remains consistent while only the input characteristics change. In the ST task, the model must instead learn new classes of unseen attacks over time. This setting is commonly known as class incremental learning (CIL) [ 21 ] which is the most challenging form of CL. Recently, many CIL methods have been proposed in speech processing. For example, bias correction-based [ 22 ] and replay-based techniques [ 23 , 24 , 25 , 26 ] store a few samples from past tasks as exemplars to prevent forgetting old knowledge. However, it is often impossible to obtain these samples. For instance, when an industry releases a pre-trained ST model, the public cannot use the private data of that industry for fine-tuning [ 27 ] . Thus, three core challenges arise when updating ST models incrementally. First, the method must handle an increasing number of target classes, not only real/fake assessments. Second, the method must work without using exemplars to protect private data. Third, it must adapt quickly to online training and rapid deployment to handle new attacks. In this work, we address the challenges of audio deepfake ST by proposing an exemplar-free CIL method called analytic source tracing , in short AnaST. Our approach is inspired by analytic learning, which views network training as learning linear layers. Specifically, for the first task, we replace back-propagation with a recursive least-squares (RLS)-like procedure, then freeze the feature extractor and update the classifier using RLS. When new attacks appear, AnaST adapts incrementally by updating the classifier in one epoch with an analytic strategy. This enables the classifier to reach the same output compared to joint training on all tasks. As a result, our model retains previous knowledge while adapting to learn new attacks without using historical examples. Our studies aim to showcase the effectiveness of the proposed AnaST against the existing baselines and also project its efficiency. To the best of our knowledge, this is the first work to apply CIL in ST. The code for implementation will be released publicly. 2 AnaST: Analytic Source Tracing 2.1 Problem Formulation Figure 1: An overview of the proposed AnaST method for Task \tau_{t} . We proceed to the class incremental learning stage, where the model adapts by analytic learning for one epoch per new dataset phase, assisted by a correlation matrix (Eq. ( 8 )) that encodes past knowledge. This process enables the model to learn new tasks while preserving previously acquired information. In this work, we examine an ST system that learns different attack categories through a sequence of tasks \left\{\tau_{1},\tau_{2},\dots,\tau_{T}\right\} . We consider this problem as a CIL scenario, where the system must recognize all attacks from each task, even as new tasks are introduced. For each task \tau_{t} , the input data (x,y) follow a distinct distribution \mathcal{D}_{t} . Our goal is to train a model f(x;\mathcal{W}) that adapts to new data while preserving its understanding of earlier tasks. Formally, we aim to minimize the cross-entropy loss across all tasks: \underset{\mathcal{W}}{\mathrm{argmin}}\sum_{t=0}^{T}\mathbb{E}_{\left(x,y\right)\sim\mathcal{D}_{t}}\left[\mathcal{L}_{\text{CE}}\left(y,f(x;\mathcal{W})\right)\right], (1) However, storing all past data is impractical due to memory costs and privacy concerns. In addition, simply fine-tuning the model on new data leads to catastrophic forgetting, causing the model to lose knowledge gained from previous tasks. 2.2 Proposed AnaST Method This section describes the AnaST method in detail, along with the initial and analytic learning stages. Our explanation follows the common ST models [ 10 , 13 ] , which include a CNN backbone as an acoustic feature extractor and a linear layer as the classifier. Figure 1 provides an overview of AnaST for task \tau_{t} . 2.2.1 Initial training feature extractor In the first stage, which is referred as initial training, we use standard back-propagation. During this step, the network is trained on the dataset \mathcal{D}_{0} of

Read the complete text