Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models

Nikita, Tao, Jiajun, Yingke, Tristan, Tianxiang, Simon, Kong Aik, Eng Siong · 2026

Read Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models by Nikita, Tao, Jiajun, Yingke, Tristan, Tianxiang, Simon, Kong Aik, Eng Siong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08179.txt.

Introduction

Kuzmin and colleagues ask what always-on speech models reveal about speakers. User audio passes through a language model backbone, yet its hidden-state leaks had not been checked. With a lazy-informed attacker under VoicePrivacy 2024, SALM-Duplex and Moshi hidden states leak speaker identity at every transformer layer. SALM-Duplex leaks worst early, Moshi evenly, and linkability climbs fast in early turns. Two Stream-Voice-Anon fixes follow, Anon-W2W and Anon-W2F, the latter lifting the error rate over threefold, near the random-chance ceiling.

Catalog record

CREATORNikita, Tao, Jiajun, Yingke, Tristan, Tianxiang, Simon, Kong Aik, Eng Siong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21963
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08179

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models Nikita, Tao, Jiajun, Yingke, Tristan, Tianxiang, Simon, Kong Aik, Eng Siong Abstract End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5 × \times relative to the discrete encoder baseline (11.2% → \rightarrow 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78--93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s). 1 1 1 Demo page: https://anonymous-569230593.github.io/ Kuzmin Zhong Deng Zhu Tsoi Cao Lui Lee Chng keywords speaker anonymization, full-duplex speech, privacy, speaker verification, speech agents † † address: 1 Nanyang Technological University, Singapore 2 A*STAR, Singapore 3 Huawei Leibniz Research Center 4 The Chinese University of Hong Kong 5 The Hong Kong Polytechnic University † † email: [email protected] 1 Introduction End-to-end (E2E) full-duplex speech dialogue systems represent a fundamental shift from turn-taking interaction to always-on, simultaneous listening and speaking. Models such as SALM-Duplex [ 7 ] , Moshi [ 4 ] , and SyncLLM [ 29 ] route raw user audio continuously through a decoder-only LLM backbone, computing hidden state representations at every transformer layer throughout the entire conversation. Unlike cascaded systems where user and agent streams are processed separately, the LLM core of these full-duplex systems maintains a persistent internal state over the user’s speech stream, capturing their voice, speaking style, and identity. 1 3 5 7 10 0 0.2 0.4 0.6 0.8 1 low privacy protection high privacy protection Dialogue Duration (turns) Speaker Privacy ( 1-\mathrm{Linkability} ) SALM-Duplex disc. (no anon) Moshi disc. (no anon) SALM-Duplex disc. + W2F Moshi disc. + W2W Figure 1: Speaker privacy ( 1-\mathrm{Linkability} ) vs. dialogue turn count. Only discrete encoder variants are shown; continuous encoder Linkability is omitted for clarity. Red lines: no anonymization; green lines: post-anonymization. Without anonymization, both systems drop into the low-privacy zone within a few turns; anonymization lifts privacy into the protected zone, though Moshi + W2W shows gradual degradation over dialogue length. This creates a privacy exposure that remains largely unexplored. Under GDPR and similar regulations, the mere presence of identifiable speaker information in model representations constitutes a compliance risk, regardless of whether external access is exploited, making it essential to audit and mitigate such leakage proactively. Probing studies on self-supervised speech models (e.g., wav2vec 2.0, HuBERT, WavLM) have established that their hidden representations strongly encode speaker identity [ 20 , 32 , 3 ] . In the text domain, prior work has shown that LLM hidden states can leak demographic and content-level user attributes [ 24 ] , and membership inference probes have exposed privacy risks in masked language models [ 14 ] . Our focus is distinct: we investigate speaker identity leakage, the persistent encoding of who is speaking , which enables re-identification regardless of conversational content. Yet, to the best of our knowledge, no prior work has examined whether these always-on LLM hidden states retain sufficient speaker identity to enable re-identification. Figure 2: Overview of the original SALM-Duplex pipeline and proposed anonymization setups. The main diagram shows the ASR-based encoder baseline: an ECAPA-TDNN probe attached to the LLM’s hidden states (red dashed path) reveals substantial speaker identity leakage. The Anon-W2W inset (upper left) prepends Stream-Voice-Anon to anonymize the waveform before the unchanged ASR encoder. The Anon-W2F inset (upper right) replaces the ASR encoder with the Stream-Voice-Anon encoder (anonymization active) and fine-tunes the LLM, eliminating the redundant waveform synthesis step. (Anon-W2F is demonstrated for SALM-Duplex; the Anon-W2W setup is additionally evaluated on Moshi.) We answer this question empirically for two prominent E2E full-duplex systems, SALM-Duplex and Moshi [ 4 ] . Following the VoicePrivacy 2024 Challenge [ 26 ] evaluation protocol with a lazy-informed attacker scenario, we train a speaker verification attacker (probe) on hidden state representations extracted from each layer group. Equal error rate (EER) serves as our primary privacy metric; we additionally report Linkability from the legally validated evaluation framework of [ 28 ] . Our mean-pooled EER for the original SALM-Duplex (continuous encoder) is 28.5%, revealing significant speaker identity leakage, while Moshi’s discrete encoder reaches 6.4% EER (near-perfect identification). We further propose and evaluate two streaming anonymization setups (Figure 2 ) using Stream-Voice-Anon [ 8 ] to mitigate this leakage without sacrificing dialogue utility. Our contributions are: • We characterise speaker identity leakage in the hidden states of two E2E full-duplex dialogue LLMs (SALM-Duplex and Moshi), extending probing methodology [ 25 , 20 ] from static encoders to always-on dialogue backbones. • We provide a layer-wise and turn-length-wise analysis showing which parts of the LLM carry the most speaker-identifying information and how leakage accumulates over dialogue length. • We propose two streaming anonymization setups: Anon-W2W applies Stream-Voice-Anon at the waveform level (retaining the original encoder, validated on both SALM-Duplex and Moshi), while Anon-W2F replaces the continuous encoder with a discrete encoder and activates anonymization at the feature level. 2 Related Work 2.1 E2E speech dialogue language models The generative spoken language model (GSLM) [ 10 ] established the paradigm of modelling speech directly from discrete units, subsequently extended to two-channel spoken dialogue by dGSLM [ 16 ] , the first fully E2E full-duplex dialogue model. SpeechGPT [ 34 ] and SpiRit-LM [ 17 ] demonstrated that a single LLM backbone can natively interleave speech and text tokens for conversational interaction. SyncLLM [ 29 ] introduced synchronised LLM inference to support real-time full-duplex turn management. The most recent systems, SALM-Duplex [ 7 ] and Moshi [ 4 ] , achieve low-latency always-on operation through continuous dual-stream processing. Crucially, because all of these models process raw user speech as input, speaker identity information may be inadvertently encoded in their LLM hidden representations, yet none has been analysed from a privacy perspective. 2.2 Speaker anonymization The VoicePrivacy Challenge [ 19 , 26 ] established standardised evaluation protocols and benchmarks for speaker anonymization, using the EER of an ASV system as the primary privacy metric; Vauquier et al. [ 28 ] recently extended this with a legally validated framework incorporating Linkability metrics aligned with GDPR requirements. Anonymization approaches include x-vector-based methods [ 12 , 19 ] , perturbation [ 9 ] , phonetic intermediate representations [ 13 ] , neural audio codec language models [ 18 , 33 ] , and disentangled speech representations [ 2 , 1 ] . Streaming anonymization systems, including Stream-Voice-Anon [ 8 ] , TVTSyn [ 23 ] , DarkStream [ 22 ] , and [ 21 ] , extend these methods to the real-time setting required by E2E full-duplex models; we select Stream-Voice-Anon as our front-end owing to its competitive privacy--utility trade-off and open-source availability 2 2 2 https://github.com/Plachtaa/StreamVoiceAnon . All of these approaches operate at the waveform level . To

Read the complete text