Audio Language Model for Deepfake Detection Grounded in Acoustic Chain-of-Thought
Runkun, Yixiong, Pengyu, Yuante, Massa, Bhiksha · 2026
Read Audio Language Model for Deepfake Detection Grounded in Acoustic Chain-of-Thought by Runkun, Yixiong, Pengyu, Yuante, Massa, Bhiksha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28021.txt.
Introduction
Chen and five co-authors present CoLMbo-DF, an audio language model that detects deepfake speech and explains its calls. Older detectors give a bare yes or no and hide their grounds. CoLMbo-DF feeds the model text summaries of low-level sound traits, such as pitch patterns and spectral cues, so its reasoning rests on stated evidence. The team built a dataset of audio pairs with step-by-step notes. Their smaller model beat larger audio language model baselines.
Catalog record
| CREATOR | Runkun, Yixiong, Pengyu, Yuante, Massa, Bhiksha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23947 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28021
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audio Language Model for Deepfake Detection Grounded in Acoustic Chain-of-Thought Runkun, Yixiong, Pengyu, Yuante, Massa, Bhiksha Abstract Deepfake speech detection systems are often limited to binary classification tasks and struggle to generate interpretable reasoning or provide context-rich explanations for their decisions. These models primarily extract latent embeddings for authenticity detection but fail to leverage structured acoustic evidence such as prosodic, spectral, and physiological attributes in a meaningful manner. This paper introduces CoLMbo-DF , a Feature-Guided Audio Language Model that addresses these limitations by integrating robust deepfake detection with explicit acoustic chain-of-thought reasoning. By injecting structured textual representations of low-level acoustic features directly into the model prompt, our approach grounds the model’s reasoning in interpretable evidence and improves detection accuracy. To support this framework, we introduce a novel dataset of audio pairs paired with chain-of-thought annotations. Experiments show that our method, trained on a lightweight open-source language model, significantly outperforms existing audio language model baselines despite its smaller scale, marking a significant advancement in explainable deepfake speech detection. Chen Fang Chang Li Baali Raj keywords audio LM, deepfake, language models, spoofing † † address: Carnegie Mellon University, USA † † email: [email protected] 1 Introduction Audio deepfake detection is a critical challenge in speech forensics, with applications in security, fraud prevention, misinformation detection, and speaker verification [ 1 ] .Traditional approaches have relied on signal-level artifacts, such as inconsistencies in pitch [ 2 , 3 ] , formant trajectories [ 4 ] , energy patterns [ 5 ] , and voice quality statistics [ 6 ] . These methods, however, are inflexible; their performance heavily depend on feature engineering, and may degrade under domain shift. More recent works have explored applying deep learning to audio deepfake detection [ 7 , 8 , 9 , 10 ] . While these methods exhibit strong performance on existing data, they offer limited interpretability, which could be problematic in high-stakes forensic settings. Recent advances in multimodal modeling have explored the use of Audio Language Models (ALM), but their application to deepfake detection remains limited. Existing Speaker Language Models (SLMs), such as CoLMbo [ 11 ] and SpeakerLM [ 12 ] , primarily focus on speaker profiling. These models typically make decisions in a single step, without explicitly articulating how acoustic evidence supports the final judgment, limiting their utility for high-stakes forensic settings where human validation and accountability are required. A key requirement in audio deepfake detection is grounding model decisions in measurable acoustic evidence. Purely implicit embeddings are high-dimensional and abstract, whereas reliable forensic reasoning demands structured, semantically grounded observations over prosodic, spectral, temporal, and voice quality cues. This grounding is critical for reliable detection but is not adequately addressed by existing ALMs, which often fail to generalize to unseen synthesizers or provide transparent decision rationales. To address these challenges, we present CoLMbo-DF , a lightweight Feature-Guided Audio Language Model designed for audio deepfake detection. By injecting structured acoustic evidence directly into the language model input, our approach trains the model to produce feature-grounded chain-of-thought reasoning before emitting its final decision. New acoustic feature types can be incorporated simply by augmenting the training data, without modifying the model architecture. CoLMbo-DF employs a projection-based architecture, where audio embeddings from a pretrained encoder are mapped into the language model’s input space. Structured textual descriptions of low-level acoustic features are injected alongside the audio prefix to ground the model’s reasoning in interpretable evidence. To support this formulation, we curate a new dataset FakeReason which consists of audio pairs with chain-of-thought annotations, built from ASVspoof 2019 [ 13 ] and VoxCeleb2 [ 14 ] , augmented with recent TTS systems. Experiments show that feature-guided reasoning improves robustness under synthesizer shift while requiring only a small base language model. CoLMbo-DF outperforms existing pretrained audio language model baselines on ASVspoof 2019 despite its smaller scale, demonstrating that better acoustic grounding outperforms larger parameter counts. In summary, our key contributions are as follows: • We propose CoLMbo-DF , a lightweight feature-guided ALLM pipeline for Audio Deepfake Detection that integrates structured acoustic reasoning into an audio language model, surpassing existing pretrained ALM baselines on ASVspoof 2019 with a smaller base model. • We curate a new dataset FakeReason of audio pairs and chain-of-thought annotations using ASVspoof 2019, VoxCeleb2, state-of-the-art TTS models, and a text-only LLM for CoT generation. 2 CoLMbo-DF Architecture Figure 1: An overview of our proposed pipeline. Given an input pair, the audios are encoded and then projected by the pretrained audio encoder \mathcal{E} and the projector module \mathcal{P} . Separately, acoustic features are extracted and exported to textual form as acoustic evidence. These are combined into the input of LLM, which uses reasoning to generate the final prediction. 2.1 Architecture overview Figure 1 presents our reasoning-centric audio-LLM pipeline. The key design is to decouple acoustic representation learning from decision reasoning: a strong pretrained speech encoder provides compact acoustic evidence, while an LLM performs structured reasoning conditioned on both latent embeddings and explicit, interpretable acoustic attributes. Our architecture contains three modules: (i) a pretrained audio encoder \mathcal{E} , (ii) a lightweight trainable projector \mathcal{P} that aligns audio representations to the LLM embedding space, and (iii) an instruction-tuned LLM as the reasoning core. We use pretrained WavLM-plus [ 15 ] model as the audio encoder. Given an utterance waveform x , the encoder produces frame-level representations \mathbf{H}=\mathcal{E}(x)\in\mathbb{R}^{T\times d_{e}} , which are pooled into a single vector \mathbf{h}=\mathrm{Pool}(\mathbf{H})\in\mathbb{R}^{d_{e}} . The projector maps \mathbf{h} to a latent embedding prefix compatible with the LLM input space: \mathbf{Z}=\mathcal{P}(\mathbf{h})\in\mathbb{R}^{m\times d_{\ell}}, where m is the number of projected tokens. We prepend \mathbf{Z} to the textual prompt so that the LLM can attend to audio evidence through standard self-attention, without modifying the LLM architecture. During inference, the model is prompted to generate outputs in a structured format to facilitate the extraction of the predicted label. 2.2 Acoustic Evidence-grounded Reasoning In addition to latent embeddings, we also extract interpretable acoustic features to assist the model reasoning. In audio forensics, analysts may use defined audio features, such as pitch and formants, to identify synthesized speech. [ 7 ] Inspired by this, we have provided the LLM with a selection of audio features for grounded reasoning. The features are serialized into a structured text block and included in the input as explicit acoustic evidence. This encourages the LLM to ground its rationale in measurable attributes rather than relying solely on opaque embeddings. The explicit evidence also acts as a mechanism for feature disentanglement, forcing the model to separately consider independent variables, such as pitch and pauses, before making a decision. We train the model via supervised fine-tuning (SFT) with next token prediction. Each training instance consists of an input audio pair (x_{1},x_{2}) and their extracted acoustic features, which are serialized and concatenated with the projected audio embeddings. The model is expected to output a chain-of-thought representing its reasoning process, followed by a structured output that includes the predicted label (real or fake). 3 FakeReason Dataset To train our model, we constructed the FakeReason dataset, consisting of audio pairs and chain-of-thought annotations regarding the authenticity of the audio. Each audio pair consists of one authentic speech as reference, and the other as the subject of classification. For the audio pairs in our dataset, we