DiffAnon: Diffusion-based Prosody Control for Voice Anonymization
Ismail Rasim, Zexin, Nicholas, Philipp, Berrak · 2026
Read DiffAnon: Diffusion-based Prosody Control for Voice Anonymization by Ismail Rasim, Zexin, Nicholas, Philipp, Berrak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26281.txt.
Introduction
Ulgen and four colleagues built DiffAnon, a voice anonymizer that lets users dial how much prosody, the melody and rhythm of speech, survives the disguise. Prosody carries meaning and feeling but also leaks identity, and older tools either drop it or fix the trade-off in advance. Using diffusion with classifier-free guidance, DiffAnon slides smoothly between stronger disguise and truer prosody inside one model. Tests showed a structured trade-off with strong utility and competitive privacy.
Catalog record
| CREATOR | Ismail Rasim, Zexin, Nicholas, Philipp, Berrak |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25523 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26281
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiffAnon: Diffusion-based Prosody Control for Voice Anonymization Ismail Rasim, Zexin, Nicholas, Philipp, Berrak Abstract To preserve or not to preserve prosody is a central question in voice anonymization. Prosody conveys meaning and affect, yet is tightly coupled with speaker identity. Existing methods either discard prosody for privacy or lack a principled mechanism to control the utility–privacy trade-off, operating at fixed design points. We propose DiffAnon, a diffusion-based anonymization method with classifier-free guidance (CFG) that provides explicit, continuous inference-time control over prosody preservation. DiffAnon refines acoustic detail over semantic embeddings of an RVQ codec, enabling smooth interpolation between anonymization strength and prosodic fidelity within a single model. To the best of our knowledge, it is the first voice anonymization framework to provide structured, interpolatable inference-time prosody control. Experiments demonstrate structured trade-off behavior, achieving strong utility while maintaining competitive privacy across controllable operating points. Ulgen Cai Andrews Koehn Sisman keywords voice anonymization, prosody, diffusion † † address: 1 Center for Language and Speech Processing, Johns Hopkins University, USA 2 Human Language Technology Center of Excellence (COE), Johns Hopkins University, USA † † email: [email protected], [email protected], [email protected], [email protected] 1 Introduction Voice anonymization aims to protect speaker privacy by concealing speaker identity in a speech signal, while still conveying the intended linguistic content for effective communication [ 27 ] . Speech jointly encodes linguistic content, para-linguistic information, and speaker identity, all of which are inherently entangled. As a result, voice anonymization faces a fundamental challenge: enhancing privacy often degrades utility, whereas preserving utility may reveal speaker identity, leading to an inherent trade-off between utility and privacy [ 35 , 2 ] . A key factor shaping the utility–privacy trade-off in voice anonymization is prosody, which conveys emphasis, emotion, and speaker intent [ 28 ] . For example, the utterance “You did that“ can express a neutral statement, an accusation, disappointment, or a question depending on its prosodic realization. Preserving prosody is therefore essential for maintaining meaning and expressiveness [ 4 , 7 ] . At the same time, prosody is closely linked to speaker identity, as speakers exhibit characteristic prosodic patterns [ 8 , 11 , 14 ] . Modern speaker recognition systems can exploit such prosodic cues, making prosody a potential source of identity leakage [ 3 , 29 ] . As a result, prosody lies at the center of the utility–privacy trade-off in voice anonymization: preserving it improves utility but can compromise privacy, while modifying or suppressing it strengthens anonymization at the cost of degraded expressiveness and meaning [ 26 , 18 , 2 ] . Existing voice anonymization approaches adopt different strategies to navigate the utility–privacy trade-off. Cascaded ASR–TTS systems [ 16 , 17 ] prioritize privacy by transcribing speech into text and resynthesizing it, effectively removing speaker-specific acoustic characteristics while discarding the source prosody, which reduces expressiveness and para-linguistic fidelity. In contrast, voice conversion (VC)–based anonymization methods [ 5 , 20 , 1 , 10 ] aim to preserve linguistic content and aspects of source prosody while modifying speaker identity, but imperfect disentanglement can lead to residual identity leakage. Some approaches additionally apply random perturbations to prosodic features [ 15 , 23 , 6 ] , obscuring identity at the cost of unpredictable utility degradation. Across these methods, prosody handling is largely fixed by design, with systems suppressing, implicitly preserving, or heuristically perturbing it. While recent work has explored combining multiple anonymization systems for flexibility [ 32 ] , they do not enable continuous, inference-time control within a single model. Consequently, most existing systems operate at largely fixed points on the utility–privacy trade-off without explicit prosody control. To address these limitations, we propose DiffAnon , a diffusion-based voice anonymization framework with classifier-free guidance (CFG) [ 9 ] that is specifically designed to enable controllable anonymization. CFG brings adjustable guidance during anonymization of speech without the need of external classifiers which are lacking for prosody. DiffAnon introduces a systematic mechanism for modifying source prosody by adjusting the strength of prosodic conditioning during inference, thereby controlling how much source prosody is preserved in anonymized speech. It formulates anonymization as an iterative refinement of acoustic details over semantic representations obtained from a residual vector quantization (RVQ)–based speech codec [ 36 ] . By refining acoustic information on top of speaker-agnostic semantic embeddings, the diffusion process aligns naturally with the iterative refinement structure of RVQ, making it well suited for controlled anonymization. This formulation allows CFG to directly regulate the contribution of source prosody during generation, enabling continuous and interpretable adjustment of the utility--privacy trade-off within a single trained model. The structured generation formulation of diffusion allows DiffAnon to retain meaningful speech utility while flexibly adjusting anonymization strength. † † DiffAnon is available at: https://github.com/lightensyrup/diffanon.git Our contributions can be summarized as: (i) we introduce DiffAnon, the first voice anonymization framework that provides explicit control over source prosody preservation via classifier-free guidance; (ii) we propose a novel diffusion-based formulation that refines acoustic information over speaker-agnostic semantic embeddings in an RVQ-based codec space; and (iii) we show on VoicePrivacy Challenge 2024 that DiffAnon achieves strong utility and competitive privacy with systematic utility–privacy trade-off navigation. We publicly release the codes and pretrained models to enable reproducibility. Figure 1: a) Conditional diffusion training to construct codec embeddings b) Anonymization inference via adjusted prosody with CFG and pseudo-speaker condition (randomly sampled from a pseudo-speaker pool) 2 The Utility–Privacy Trade-off Voice anonymization is inherently a multi-objective problem, as speech simultaneously encodes linguistic content, speaker identity, prosody, and emotion. As a result, anonymization must be evaluated along competing dimensions rather than a single metric. The VoicePrivacy initiative [ 27 , 18 , 26 ] formalizes this utility–privacy trade-off through standardized evaluation protocols. Privacy is measured as resistance to speaker verification attacks under different attacker knowledge assumptions, typically quantified by equal error rate (EER) [ 12 ] . Utility is assessed through linguistic preservation using word error rate (WER), and para-linguistic preservation using task-specific metrics such as emotion recognition recall (UAR) in VoicePrivacy 2024 [ 26 ] and F0 correlation in VoicePrivacy 2022 [ 18 ] . Across challenge editions, results consistently show that stronger anonymization often degrades prosodic and emotional fidelity, while better preservation of para-linguistic information increases the risk of speaker re-identification [ 18 , 35 , 26 , 2 ] . Consequently, most existing approaches operate at mostly fixed design points on this trade-off, lacking mechanisms for continuous, inference-time adjustment. In contrast, our approach enables explicit and controllable navigation of the utility–privacy trade-off through prosody control within a single model. 3 DiffAnon We introduce DiffAnon, a diffusion-based framework for controllable voice anonymization (Fig. 1 ). The model is trained to reconstruct SpeechTokenizer codec embeddings [ 36 ] conditioned on separately extracted representations of linguistic content, prosody, and speaker identity, which are then decoded into waveform via the codec decoder. At inference, DiffAnon anonymizes speech by preserving source content and prosody while replacing the original speaker identity with a pseudo-speaker embedding. The degree of prosody preservation is regulated through classifier-free guidance (CFG), enabling explicit and continuous control of anonymization strength. Content Condition . As the content condition, DiffAnon utilizes the first-level quantization embeddings of SpeechTokenizer [ 36 ] , an RVQ-based neural speech codec. SpeechTokenizer encodes speech into multi-level tokens