StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation

Nikita, Kong Aik, Eng Siong · 2026

Read StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation by Nikita, Kong Aik, Eng Siong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06079.txt.

Introduction

Kuzmin, Lee, and Chng tackle a weakness of streaming speaker anonymization: systems that hide identity often flatten emotion. Codec language models trained to continue audio discard emotional cues and default to dominant acoustic patterns instead of paralinguistic detail. The team fine-tunes only, pairing neutral-emotion utterances from the same speaker with frame-level emotion distillation over acoustic token hidden states. On the VoicePrivacy 2024 protocol the method reaches 49.2% UAR for emotion preservation, a relative gain of 24% over the baseline, while intelligibility and privacy remain strong.

Catalog record

CREATORNikita, Kong Aik, Eng Siong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21773
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06079

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation Nikita, Kong Aik, Eng Siong Abstract We address the challenge of preserving emotional content in streaming speaker anonymization (SA). Neural audio codec language models trained for audio continuation tend to degrade source emotion: content tokens discard emotional information, and the model defaults to dominant acoustic patterns rather than preserving paralinguistic attributes. We propose supervised finetuning with neutral-emotion utterance pairs from the same speaker, combined with frame-level emotion distillation on acoustic token hidden states. All modifications are confined to finetuning, which takes less than 2 hours on 4 GPUs and adds zero inference latency overhead, while maintaining a competitive 180ms streaming latency. On the VoicePrivacy 2024 protocol, our approach achieves a 49.2% UAR (emotion preservation) with 5.77% WER (intelligibility), a +24% relative UAR improvement over the baseline (39.7% → \to 49.2%) and +10% over the emotion-prompt variant (44.6% UAR), while maintaining strong privacy (EER 49.0%). 1 1 1 Demo and code: https://anonymous3842031239.github.io Kuzmin Lee Chng keywords speaker anonymization, emotion preservation, streaming speech processing, knowledge distillation † † address: 1 Nanyang Technological University, Singapore 2 Institute for Infocomm Research (I2R), A*STAR, Singapore 3 The Hong Kong Polytechnic University, Hong Kong † † email: [email protected] 1 Introduction Speaker anonymization (SA) aims to transform an input speech (source) to conceal speaker identity while preserving linguistic content and paralinguistic attributes such as emotion [ 30 , 21 , 29 ] . Real-time SA is essential for privacy-preserving applications including teleconferencing, call centers, voice assistants, and online mental health counseling [ 18 ] , where low latency is critical. While privacy and intelligibility have been the primary focus of SA research, emotion preservation is equally important for natural communication. Losing emotional cues can significantly degrade user experience and communication effectiveness in real-world deployments [ 3 ] . Recent advances in neural audio codec (NAC) language models have enabled streaming SA with competitive privacy-intelligibility trade-offs [ 20 , 13 ] , unlike offline approaches that require full utterance context [ 33 , 32 ] . These models process speech as interleaved content and acoustic token sequences, allowing autoregressive (AR) generation with controllable speaker identity. However, two fundamental limitations emerge for emotion preservation: (1) the audio continuation training paradigm encourages the model to degrade emotion preservation for the source utterance, and (2) the VQ bottleneck in neural codecs discards fine-grained acoustic details that carry emotional information [ 34 , 8 ] . To mitigate this limitation, prior work [ 13 ] employed multiple emotion-diverse prompts at inference, which partially improves emotion preservation but significantly degrades intelligibility and requires emotion-labeled prompts that are harder to obtain compared to diverse neutral prompts commonly used for anonymization. We propose a supervised finetuning (SFT) approach with frame-level emotion distillation (Figure 2 ) that addresses the root cause of emotion degradation. Our contributions are: • We show that emotion degradation in NAC-based streaming SA is primarily a training paradigm problem, not a model capacity issue: restructuring training pairs yields 3 \times larger gains than adding emotional data alone (ablation: Exp1 vs Exp2). • We apply frame-level emotion distillation to acoustic token hidden states, isolating emotion learning from content supervision (next-token prediction) to avoid gradient competition. Ablation confirms this design choice: acoustic-branch distillation outperforms semantic-branch in both UAR and WER (Exp6 vs Exp7). • Our approach achieves the highest reported emotion preservation among streaming SA methods (49.2% UAR, +24% relative over baseline) with zero inference overhead and improved privacy (Figure 1 ). Figure 1: Privacy-emotion trade-off for streaming speaker anonymization methods. Our method (orange star) compared to prior streaming methods (triangles). Figure 2: Training and inference configurations. Training : prompt and source share the same speaker but differ in emotion, forcing the model to generate emotional output from source content rather than copying prompt-specific patterns. Frame-level emotion distillation ( \mathcal{L}_{\text{emo}} , dashed) on Slow AR acoustic hidden states provides additional learning signal. Inference : a neutral utterance from the target anonymous speaker conceals source identity while the finetuned model preserves source emotion; no latency is added over the baseline. The transformer f_{\theta} is depicted twice for visual clarity; weights are shared. 2 Related Work Speaker Anonymization. Traditional SA methods modify speech features such as pitch and formants [ 22 , 21 ] or replace speaker embeddings [ 28 , 27 , 14 , 17 ] . Recent neural approaches use voice conversion [ 16 , 33 ] or NAC-based language models [ 20 , 13 ] . Most methods prioritize privacy-intelligibility trade-offs, with emotion preservation receiving less attention [ 8 ] . Streaming SA and Voice Conversion. Real-time SA and VC systems face a fundamental trade-off between latency and quality. StreamVoice [ 31 ] introduced NAC-based language models for streaming VC with interleaved semantic and acoustic token generation, achieving low latency through autoregressive processing. A key limitation identified in recent work [ 26 ] is the temporal mismatch between static speaker embeddings and dynamic content representations. While content evolves frame-by-frame, conventional systems inject speaker identity as a global vector, leading to over-smoothed timbre and reduced expressivity. Time-varying timbre (TVT) approaches address this by allowing speaker characteristics to vary synchronously with content through attention-based memory mechanisms. Self-supervised approaches like GenVC [ 2 ] eliminate dependency on external speaker encoders through autoregressive generation. These architectural insights are complementary to our supervised finetuning approach, which addresses emotion degradation from a training paradigm perspective. Emotion in SA. Speaker anonymization systems often degrade emotional content [ 9 , 6 , 12 ] , with the primary cause identified as information loss in intermediate representations [ 8 ] . Several approaches address this through emotion embeddings [ 19 ] or prosody-aware anonymization [ 10 ] . EASY [ 33 ] introduced emotion-aware offline SA through factorized distillation, achieving strong emotion preservation (63.8% UAR) but requiring full utterance context. For streaming SA, StreamVoiceAnon [ 13 ] showed limited emotion preservation (39.7% UAR) with neutral prompts. Prior work [ 24 ] achieved low latency but weak privacy (31.4% EER), while DarkStream [ 25 ] operates at \SI 200 \milli \second latency but shows limited emotion preservation (34.7% UAR). Recent streaming VC work [ 26 ] demonstrates that architectural improvements can improve naturalness, but these systems still face emotion challenges where content bottlenecks and speaker replacement create additional pressure on paralinguistic retention. Knowledge Distillation for Speech. Knowledge distillation has been widely applied in speech processing to transfer representations from large pretrained models to smaller task-specific models [ 5 ] . Self-supervised models capture emotion-relevant information in their hidden representations [ 23 ] ; Emotion2Vec+ [ 15 ] provides frame-level emotion representations through pretraining on large-scale speech data. Unlike utterance-level emotion labels, these frame-level features capture fine-grained emotional dynamics at each time step, making them suitable for streaming applications. We leverage such features as distillation targets, carefully selecting the acoustic branch to avoid interference with content supervision on the semantic branch. 3 Proposed Method 3.1 Problem Statement Streaming anonymization models generally struggle with emotion preservation due to information bottlenecks and training objectives that prioritize content and speaker conversion over paralinguistic attributes. In NAC-based approaches specifically, we identify two key issues: (1) the audio continuation training paradigm encourages the model to degrade emotion preservation for the

Read the complete text