Toward Fine-Grained Speech Inpainting Forensics: A Dataset, Method, and Metric for Multi-Region Tampering Localization

Tung Vu, Yen Nguyen, Hai Nguyen, Cuong Pham, Cong Tran · 2026

Read Toward Fine-Grained Speech Inpainting Forensics: A Dataset, Method, and Metric for Multi-Region Tampering Localization by Tung Vu, Yen Nguyen, Hai Nguyen, Cuong Pham, Cong Tran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02223.txt.

Introduction

Tung Vu and colleagues tackle speech tampering where a few words inside a recording are replaced to change its meaning while keeping the speaker's voice. Voice cloning and speech synthesis make such partial edits realistic. Past benchmarks focus on whole-utterance fakes or a single tampered region, leaving multi-region localisation untested.

They introduce MIST, a large multilingual data set in six languages with one to three word-level inpainted segments per utterance, made by guided semantic replacement and neural cloning. Fake content is only a small share of each file. Their method, ISA, scans in a coarse-to-fine sliding window and works with varied backbones. A new metric scores fine-grained localisation across regions whose count is unknown in advance.

Catalog record

CREATORTung Vu, Yen Nguyen, Hai Nguyen, Cuong Pham, Cong Tran
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Toward Fine-Grained Speech Inpainting Forensics: A Dataset, Method, and Metric for Multi-Region Tampering Localization Tung Vu, Yen Nguyen, Hai Nguyen, Cuong Pham, Cong Tran Abstract Recent advances in voice cloning and text-to-speech synthesis have made partial speech manipulation—where an adversary replaces a few words within an utterance to alter its meaning while preserving the speaker’s identity—an increasingly realistic threat. Existing audio deepfake detection benchmarks focus on utterance-level binary classification or single-region tampering, leaving a critical gap in detecting and localizing multiple inpainted segments whose count is unknown a priori. We address this gap with three contributions. First, we introduce MIST ( M ulti-region I npainting S peech T ampering), a large-scale multilingual dataset spanning 6 languages with 1–3 independently inpainted word-level segments per utterance, generated via LLM-guided semantic replacement and neural voice cloning, with fake content constituting only 2–7% of each utterance. Second, we propose ISA ( I terative S egment A nalysis), a backbone-agnostic framework that performs coarse-to-fine sliding-window classification with gap-tolerant region proposal and boundary refinement to recover all tampered regions without prior knowledge of their count. Third, we define SF1@ τ \tau , a segment-level F1 metric based on temporal IoU matching that jointly evaluates region count accuracy and localization precision. Zero-shot evaluation reveals that partial inpainting at word granularity remains unsolved by existing deepfake detectors: utterance-level classifiers trained on fully synthesized speech assign near-zero fake probability to MIST utterances where only 2--7% of content is manipulated. ISA consistently outperforms non-iterative baselines in this challenging setting, and the dataset, code, and evaluation toolkit are publicly released. 1 1 1 https://huggingface.co/datasets/tung2308/MIST_SpeechInpaintingDataset 1 Introduction The proliferation of neural text-to-speech (TTS) and voice conversion (VC) technologies has given rise to increasingly sophisticated audio deepfakes 20 . While fully synthesized speech has received extensive research attention, a more insidious form of manipulation— partial speech inpainting —poses a uniquely dangerous threat. In this scenario, an adversary replaces only a few carefully chosen words within a genuine utterance, preserving the original speaker’s voice characteristics, prosody, and recording conditions for the vast majority of the signal. By changing as few as one to three semantically critical words, the meaning of a statement can be drastically altered (e.g., “I support this policy” \rightarrow “I oppose this policy”) while remaining nearly imperceptible to human listeners. Unlike fully synthesized speech, the fake content in a partial inpainting attack constitutes only 2–7% of the utterance duration, making it orders of magnitude harder to detect and precisely localize. Existing audio deepfake detection systems have achieved remarkable progress on the fully-synthesized speech scenario. Utterance-level classifiers such as RawNet2 12 and AASIST 7 operate directly on raw waveforms or spectro-temporal graphs to produce a single binary real/fake decision. Self-supervised approaches leveraging Wav2Vec 2.0 1 and WavLM 4 features have further pushed performance on ASVspoof benchmarks 15 ; 10 ; 17 . For partial manipulation, PartialSpoof 21 introduced multi-resolution countermeasures for simultaneous utterance- and segment-level detection, while Half-Truth 18 , LAV-DF 3 , and LlamaPartialSpoof 9 explored single-region splicing at increasing scale. Despite these advances, all existing approaches share a common assumption: each utterance contains at most one contiguous tampered region, and its presence is confirmed by a binary utterance-level label. This assumption breaks down precisely in the most realistic and dangerous attack scenario: an adversary who replaces multiple scattered words to alter the conveyed message. Three interrelated gaps prevent existing work from addressing this threat. Dataset gap: no publicly available benchmark provides utterances with more than one independently inpainted region, multilingual coverage, or word-level temporal annotations for each fake segment. Methodological gap: even methods that attempt temporal localization assume a fixed or known number of tampered regions; when the manipulation count is unknown a priori, frame-level approaches produce fragmented predictions with no principled aggregation into coherent segment hypotheses. Evaluation gap: standard metrics—utterance-level accuracy, equal error rate (EER), or frame-level AUC—penalize neither over-segmentation nor under-segmentation and thus fail to capture the dual challenge of correctly counting tampered regions and precisely localizing their temporal boundaries. Our zero-shot experiments confirm the severity of the methodological gap: state-of-the-art utterance-level deepfake classifiers assign near-zero fake probability to utterances where only 2–7% of content is manipulated, regardless of the inference-time strategy applied. To address these gaps, we make the following contributions: 1. MIST Dataset (Section 3 ): A large-scale multilingual benchmark spanning five languages (Chinese, English, French, Japanese, Vietnamese) with 1–3 independently inpainted word-level segments per utterance, generated via LLM-guided semantic replacement and state-of-the-art neural voice cloning. MIST contains 598k utterances (478 h genuine, 1,119 h inpainted) and provides precise word-level temporal annotations for every fake segment, making it the first dataset to systematically evaluate multi-region partial inpainting detection. 2. ISA Method (Section 4 ): An Iterative Segment Analysis pipeline that performs coarse-to-fine sliding-window classification, gap-tolerant region proposal merging, and boundary refinement to localize all tampered regions without requiring prior knowledge of their count. ISA is backbone-agnostic and introduces no additional trainable parameters beyond the underlying classifier. 3. SF1@ \tau Metric (Section 5 ): A segment-level F1 score based on temporal IoU matching that jointly evaluates region count accuracy and localization precision in a single interpretable number, complemented by Count Accuracy (CA) to disentangle counting errors from boundary errors. Experiments demonstrate that ISA consistently outperforms frame-level and single-window baselines even in the zero-shot regime, and our analysis establishes the first quantitative evidence that partial inpainting at word granularity is an open, unsolved problem for the audio forensics community. The MIST dataset, ISA codebase, and SF1@ \tau evaluation toolkit are publicly released to accelerate progress on this challenge. 2 2 2 https://huggingface.co/datasets/tung2308/MIST_SpeechInpaintingDataset 2 Related Work 2.1 Audio Deepfake Detection Audio deepfake detection has been extensively studied in the context of automatic speaker verification (ASV) spoofing countermeasures. The ASVspoof challenge series 15 ; 10 ; 16 has driven progress through standardized benchmarks covering TTS, VC, and replay attacks. State-of-the-art systems include RawNet2 12 , which operates on raw waveforms; AASIST 7 , which combines spectral and temporal graph attention; and self-supervised approaches leveraging Wav2Vec 2.0 1 and WavLM 4 features for spoofing detection 13 . However, these methods produce utterance-level binary decisions and are not designed to localize tampered regions within partially manipulated speech. 2.2 Partial Speech Manipulation and Datasets Partial manipulation—where only a segment of an utterance is replaced—has gained attention as a realistic attack vector. The PartialSpoof dataset 21 ; 22 introduced utterances with a single contiguous spliced region and segment-level labels. The Half-Truth dataset 19 combined real and synthetic segments at utterance boundaries. More recently, LAV-DF 3 used a rule-based system to replace words with antonyms, and AV-Deepfake1M 2 employed ChatGPT to alter sentences. LlamaPartialSpoof 8 demonstrated LLM-driven partial manipulation at scale. Negroni et al. 11 analyzed the impact of splicing artifacts in partially fake speech. While these works represent important steps, they are limited to single-region tampering or use only one or two TTS models. In contrast, real-world adversaries may replace multiple short words scattered across an utterance, a scenario not covered by existing datasets. 2.3 Tampering Localization Temporal localization of audio tampering has been approached through frame-level classification 21 , boundary detection, and attention-based methods. The PartialSpoof work 21 proposed multi-resolution countermeasures for simultaneous utterance- and segment-level detection. Most methods assume a known or

Read the complete text