TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization

Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah & Ricardo Gutierrez-Osuna · 2026

Read TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization by Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah & Ricardo Gutierrez-Osuna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09389.txt.

Introduction

Live voice conversion and speaker anonymization must run causally with low delay, yet most systems inject speaker identity as one fixed global embedding while content changes frame by frame. Waris Quamer and three co-authors introduce a streamable synthesizer in their 2026 paper that matches identity to content in time, through a time-varying timbre representation. A global timbre memory expands one timbre instance into compact facets that frame-level content can attend to.

A gate regulates how much the timbre varies, spherical interpolation keeps identity geometry smooth during local changes, and a factorized quantized bottleneck curbs leftover speaker leakage in the content stream. The full system streams end to end with latency well under a tenth of a second on graphics hardware. Trials show gains in naturalness, speaker transfer, and anonymization over leading streaming baselines, framing time-varying timbre as a scalable route to private, expressive speech.

Catalog record

CREATORWaris Quamer, Mu-Ruei Tseng, Ghady Nasrallah & Ricardo Gutierrez-Osuna
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09389

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah & Ricardo Gutierrez-Osuna Abstract Real-time voice conversion and speaker anonymization require causal, low-latency synthesis without sacrificing intelligibility or naturalness. Current systems have a core representational mismatch: content is time-varying, while speaker identity is injected as a static global embedding. We introduce a streamable speech synthesizer that aligns the temporal granularity of identity and content via a content-synchronous, time-varying timbre (TVT) representation. A Global Timbre Memory expands a global timbre instance into multiple compact facets; frame-level content attends to this memory, a gate regulates variation, and spherical interpolation preserves identity geometry while enabling smooth local changes. In addition, a factorized vector-quantized bottleneck regularizes content to reduce residual speaker leakage. The resulting system is streamable end-to-end, with < < 80 ms GPU latency. Experiments show improvements in naturalness, speaker transfer, and anonymization compared to SOTA streaming baselines, establishing TVT as a scalable approach for privacy-preserving and expressive speech synthesis under strict latency budgets. 1 Introduction Real-time voice conversion (VC) and speaker anonymization (SA) aim to deliver natural, intelligible speech while meeting strict streaming and latency constraints. Beyond words, voice recordings carry biometric and paralinguistic cues–identity, sex, age, accent, and emotion–that adversaries can exploit for recognition and profiling, creating real risks to privacy. As voice interfaces proliferate and privacy rules tighten, protecting these speech attributes without degrading the communicative utility of speech has become essential, as exemplified by several initiatives. As an example, starting in 2020 the Voice Privacy Challenge ( 41 ) evaluates speech anonymization systems on both privacy and usefulness, with shared benchmarks that quantify speaker obfuscation alongside speech quality. Further, federal agencies have pushed for the development of real-time solutions with tight latency budgets (e.g., IARPA’s Anonymous Real-Time Speech program 1 1 1 https://www.iarpa.gov/research-programs/arts ). To address this challenge, recent speech architectures have achieved sub-second latency by combining lightweight content encoders with direct waveform decoders ( 29 ; 30 ; 48 ; 5 ) . Yet a core limitation of these approaches persists: while content is represented as a time-varying sequence, speaker identity is typically injected as a single static vector. This dynamic-static mismatch dampens expressivity and often yields over-smoothed timbre, especially when articulation, emotion, or emphasis change within an utterance. Making content strongly speaker-independent (e.g., via aggressive bottlenecks) can improve anonymization performance but suppress meaningful variations in speech such as accent and emotional color, or introduce artifacts ( 30 ) . We contend this trade-off is largely architectural: a stationary speaker vector forces the decoder to reconcile incompatible time scales. A better formulation would add temporal granularity to speaker conditioning to match that of the content, while allowing control and meeting tight latency constraints. We propose TVTSyn, a streaming speech synthesizer that replaces static speaker embeddings with a time-varying timbre (TVT) representation that is synchronized with the content. A Global Timbre Memory (GTM) expands a global timbre seed into a compact set of timbre facets; frame-level content attends to this memory to retrieve the most relevant facets over time; a learned gate regulates how much timbre is allowed to vary; and spherical interpolation blends global and time-varying paths to preserve identity geometry while enabling smooth local variation. This TVT stream conditions a causal decoder alongside pitch/energy predictors, yielding natural variation while retaining control. Complementing this, a factorized vector-quantized bottleneck regularizes the content network to reduce residual identity cues while preserving linguistic content. TVTSyn runs in a streaming fashion with small, mask-based future access in the encoder and fully causal decoding. The system can generate synthesis with <80ms latency on a modern GPU and runs within a few hundred ms. latency on CPUs 2 2 2 See hardware specs in section 4 . . We evaluate the model for both VC and SA tasks under the VoicePrivacy Challenge (VPC) 2024 protocol, reporting equal-error-rates (EER) in automatic speaker verification (ASV) as a measure of privacy preservation, and word error rates (WER) of an automatic speech recognizer (ASR) as a measure of utility, as well as latency and real-time factors. Our main contributions are: • Content-synchronous timbre modeling : We introduce a time-varying timbre formulation that aligns speaker conditioning with frame-level content, resolving the static–dynamic mismatch responsible for degraded quality in streaming VC and SA. • A streamable low-latency architecture : We design a fully causal system that integrates GTM-based timbre, factorized VQ bottlenecks, and prosodic predictors, and maintains low latency while balancing naturalness with speaker fidelity, and anonymization robustness. • Comprehensive benchmarking : We evaluate across VC and anonymization tasks with perceptual quality, speaker similarity, privacy (EER), utility (WER), and runtime performance, demonstrating superior privacy–utility trade-offs over prior streaming systems. 3 3 3 Audio samples can be found at: https://anonymized0826.github.io/TVTSyn/ 2 Related Work Voice conversion (VC). Conventional VC models decompose the speech signal into content and speaker channels, then re-synthesize the content channel with a target voice identity. Early pipelines relied on cascaded ASR-TTS (text-to-speech) modules or used phonetic posteriorgram (PPG) representations to obtain a speaker-independent content stream before conditioning a multi-speaker decoder on the target voice ( 8 ; 16 ) . To avoid the brittleness of text/PPG representations, later “disentanglement” models learned content directly from audio via information bottlenecks and normalization/MI penalties, or with discrete units (e.g., VQ, HuBERT pseudo-labels) that suppress residual speaker cues while keeping phonetic detail ( 2 ; 4 ; 45 ; 28 ; 31 ) . Self-supervised speech representations such as HuBERT have since become standard for providing robust, transcript-free supervision of content features ( 43 ) . Streaming VC introduces strict latency constrains. Supervised systems such as LLVC ( 34 ) can achieve latencies as low as 20 ms, but their reliance on parallel data makes them difficult to scale. By contrast, most unsupervised VC approaches adopt causal encoders with small future peeks, sliding-window inference, and fast waveform decoders, achieving sub-second latency while maintaining intelligibility ( 29 ; 30 ; 48 ; 5 ; 51 ) . Here, speaker identity is usually a single global embedding injected at every frame–by concatenating content and speaker embeddings, and more recently via FiLM/AdaIN or conditional layer normalization (CLN), which allow each channel to be scale/shift modulated, enabling few/zero-shot adaptation in multi-speaker TTS/VC ( 9 ; 27 ) . A complementary line models intra-speaker variability with multi-center (sub-center) training, learning multiple anchors per speaker to reduce intra-class scatter and capture channel/phonetic/affective variation ( 42 ) . However, these streaming approaches typically use static speaker embeddings that remain constant across all frames. While this enables low latency, it creates a representational mismatch with dynamic content embeddings. Recent attention-based methods attempt to address this: FreeVC ( 15 ) uses static embeddings lacking temporal variation; GenVC ( 1 ) employs learned queries but requires non-causal inference; DAFMSVC ( 3 ) enables content-aware modulation but is offline-only and lacks learnable speaker prototypes. Our GTM differs by introducing learnable prototype parameters that capture universal timbre characteristics across speakers, while speaker-specific modulation adapts them to individual identities—enabling efficient generalization under streaming constraints. Speaker anonymization (SA). Anonymization aims to mask speaker identity while preserving communicative utility. Traditional digital-signal-processing (DSP) approaches manipulate the signal through formant shifting (e.g., McAdams coefficient ( 19 ) ), frequency warping, vocal-tract length normalization,

Read the complete text