Vision Graph Non-Contrastive Learning for Audio Deepfake Detection with Limited Labels

Falih Gozi Febrinanto 1 2 , Kristen Moore 2 , Chandra Thapa 2 , Jiangang Ma 1 , Vidya Saikrishna 1 , Feng Xia 3 (🖂) · 2025

Read Vision Graph Non-Contrastive Learning for Audio Deepfake Detection with Limited Labels by Falih Gozi Febrinanto 1 2 , Kristen Moore 2 , Chandra Thapa 2 , Jiangang Ma 1 , Vidya Saikrishna 1 , Feng Xia 3 (🖂) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04942.txt.

Introduction

Falih Gozi Febrinanto and colleagues build a fake-voice detector that learns mostly without labels. Synthetic speech now sounds real enough to fool people, so detectors matter. Graph-based detectors work well but hunger for labeled examples. Labels are slow and costly to gather for audio.

Their system, SIGNL, turns each clip into two graphs, one over frequency and one over time. Graph encoders learn from unlabeled clips by matching views of the same sound, with no negative examples. A small labeled set then fine-tunes the trained encoders for the final call. The dual graph design fits audio's twin structure of pitch and rhythm. Pretraining this way captures traits that separate real speech from fakes. The authors present SIGNL as a practical route where labeled fakes are few.

Catalog record

CREATORFalih Gozi Febrinanto 1 2 , Kristen Moore 2 , Chandra Thapa 2 , Jiangang Ma 1 , Vidya Saikrishna 1 , Feng Xia 3 (🖂)
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-04942

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision Graph Non-Contrastive Learning for Audio Deepfake Detection with Limited Labels Falih Gozi Febrinanto 1 2 , Kristen Moore 2 , Chandra Thapa 2 , Jiangang Ma 1 , Vidya Saikrishna 1 , Feng Xia 3 (🖂) Abstract Recent advancements in audio deepfake detection have leveraged graph neural networks (GNNs) to model frequency and temporal interdependencies in audio data, effectively identifying deepfake artifacts. However, the reliance of GNN-based methods on substantial labeled data for graph construction and robust performance limits their applicability in scenarios with limited labeled data. Although vast amounts of audio data exist, the process of labeling samples as genuine or fake remains labor-intensive and costly. To address this challenge, we propose SIGNL 1 1 1 We will make our implementation code publicly available upon acceptance of this paper. ( S patio-temporal v I sion G raph N on-contrastive L earning), a novel framework that maintains high GNN performance in low-label settings. SIGNL constructs spatio-temporal graphs by representing patches from the audio’s visual spectrogram as nodes. These graph structures are modeled using vision graph convolutional (GC) encoders pre-trained through graph non-contrastive learning, a label-free that maximizes the similarity between positive pairs. The pre-trained encoders are then fine-tuned for audio deepfake detection, reducing reliance on labeled data. Experiments demonstrate that SIGNL outperforms state-of-the-art baselines across multiple audio deepfake detection datasets, achieving the lowest Equal Error Rate (EER) with as little as 5% labeled data. Additionally, SIGNL exhibits strong cross-domain generalization, achieving the lowest EER in evaluations involving diverse attack types and languages in the In-The-Wild dataset. I Introduction Recent advancements in audio deepfake technologies, such as text-to-speech (TTS) algorithms [ 1 , 2 ] and voice conversion techniques [ 3 , 4 ] , have significantly enhanced the quality of synthetic speech. These technologies have enabled diverse applications, including audiobook services, voice assistants, and personalized content creation [ 5 ] . However the rise of synthetic audio has also introduced critical cybersecurity risks, such as spoofing attacks against automatic speaker verification (ASV) systems and scams targeting individuals [ 6 , 7 ] . These threats necessitate the development of robust methods for detecting audio deepfakes to mitigate malicious exploitation. Graph Neural Networks (GNNs) for Audio Deepfake Detection . Among deep learning techniques used for audio deepfake detection–such as convolutional neural networks (CNN) [ 8 , 9 ] , deep residual networks (ResNet) [ 10 , 11 ] , and transformers [ 12 ] –GNNs have emerged as a highly effective approach [ 13 , 14 ] . By modeling frequency sub-bands and temporal segments as graph structures, GNNs can capture irregular and complex relationshps, enabling the detection of subtle deepfake artifacts that may span multiple frequency sub-bands and temporal segments [ 15 , 13 , 14 ] . Early implementations relied on visual representations of audio data (eg. spectrograms) segmented into patches, which formed the basis of graph construction [ 13 , 16 , 17 ] . Recent advancements have introduced more sophisticated techniques, such as integrating heterogeneous spectral and temporal sub-graphs [ 14 ] and hybrid frameworks combining learnable visual representations with GNN classifiers [ 18 , 19 ] . Challenges with GNNs . Despite their effectiveness, GNN-based approaches for audio deepfake detection face critical challenges due to their heavy reliance on labeled data for graph construction and training [ 13 , 14 ] . However, labeled data is often scarce [ 20 , 7 ] , resulting in overfitting and poor generalization, particularly when the model encounters new attack types or datasets [ 6 ] . Furthermore, the process of manually labeling audio data is both costly and time-intensive [ 21 ] , resulting in reduced performance of supervised GNN-based models in real-world scenarios due to insufficient labeled training data [ 22 ] . Although data augmentation techniques, such as adding background noise or codec transformations [ 23 , 24 ] , aim to enhance diversity, these modifications often fail to introduce meaningful variations in graph structures [ 25 ] . Opportunities in non-contrastive learning. Self-supervised learning (SSL) offers a promising way to leverage large amounts of unlabeled data, reducing reliance on costly human-annotations [ 22 ] . Among SSL techniques, non-contrastive learning has shown particular promise in the computer vision domain [ 26 , 27 ] . Unlike contrastive learning, non-contrastive methods do not rely on negative samples for pre-training, avoiding the risk of false negatives that can push similar samples apart [ 28 ] . Non-contrastive methods offer several additional advantages: by generating positive pairs through augmentations, they enhance data diversity, addressing the challenges associated with limited labeled datasets. Furthermore, non-contrastive learning enables a self-supervised pre-training process, allowing models to learn robust representations that can be fine-tuned with minimal labeled data, significantly reducing dependence on annotations. These methods have achieved competitive performance across various edge, node, and graph-level tasks [ 29 , 30 ] , but require domain-specific adaptations to construct meaningful graph structures, embeddings and augmentations for audio data [ 29 , 31 ] . Our Contribution. In this work, we present SIGNL ( S patio-temporal v I sion G raph N on-contrastive L earning), a novel framework designed to address the challenges posed by limited labeled data in audio deepfake detection. In contrast to existing GNN-based approaches that predominantly focus on building robust classifiers [ 19 , 13 ] , SIGNL prioritises improving graph embedding quality, thereby enhancing performance in label-scarce scenarios. SIGNL transforms audio data into spatio-temporal graphs by segmenting visual representations (e.g., spectrograms) into spatial and temporal patches, which are represented as nodes with feature vectors. These nodes are used to form separate spatial and temporal graph structures that effectively capture intricate inter dependencies within the data. The framework employs vision graph convolution (GC) encoders pre-trained using non-contrastive learning, a strategy that maximises the similarity between augmented positive graph pairs. This eliminates the reliance on negative samples while achieving performance comparable to traditional contrastive methods for graph-level tasks [ 29 ] . By learning rich representations from diverse augmented graph views during pretraining, SIGNL equips the encoders to be fine-tuned effectively for audio deepfake detection, even in scenarios with limited labeled data. The key contributions of this work are as follows: • We introduce the SIGNL framework, a novel spatio-temporal vision graph non-contrastive learning method that effectively models inter-frequency and temporal dependencies, enabling robust audio deepfake detection, even in label-scarce scenarios. • We propose the utilization of positive pairs of spatio-temporal graph data , constructed from visual representations of audio and processed through graph-based augmentations. This approach highlights the importance of increasing graph structure diversity during label-free pre-training, which facilitates the discovery of meaningful data representations and improving detection performance, particularly during fine-tuning. • We conduct an extensive evaluation of SIGNL on multiple audio deepfake datasets under limited label conditions. Our results demonstrate that SIGNL outperforms state-of-the-art supervised audio deepfake detection models as well as contrastive and non-contrastive learning models for audio data. SIGNL achieves the lowest Equal Error Rate (EER) with as little as 5% labeled data and exhibits strong generalization capabilities, consistently achieving the lowest EER in cross-domain evaluations involving different attack types and languages on the In-The-Wild dataset. The rest of this paper is organized into the following

Read the complete text