Fed-PISA: Federated Voice Cloning via Personalized Identity-Style Adaptation
Qi Wang, Shituo Ma, Guoxin Yu, Hanyang Peng, Yue Yu · 2025
Read Fed-PISA: Federated Voice Cloning via Personalized Identity-Style Adaptation by Qi Wang, Shituo Ma, Guoxin Yu, Hanyang Peng, Yue Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16010.txt.
Introduction
Cloning a voice from a few clips can make speech tools feel personal, but sharing voice data raises privacy risks. Qi Wang and four co-authors propose Fed-PISA, a method that trains voice cloning across many devices without pooling raw data. Each device keeps the speaker's tone traits at home and sends only a small style part to a shared server.
A matching step, drawn from methods that pair like users, builds a custom model for each client from peers with like styles. Tests show gains in style, natural sound, and speaker likeness over standard shared training. The gains come with very low data transfer costs.
Catalog record
| CREATOR | Qi Wang, Shituo Ma, Guoxin Yu, Hanyang Peng, Yue Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-16010
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fed-PISA: Federated Voice Cloning via Personalized Identity-Style Adaptation Qi Wang, Shituo Ma, Guoxin Yu, Hanyang Peng, Yue Yu Abstract Voice cloning for Text-to-Speech (TTS) aims to generate expressive and personalized speech from text using limited data from a target speaker. Federated Learning (FL) offers a collaborative and privacy-preserving framework for this task, but existing approaches suffer from high communication costs and tend to suppress stylistic heterogeneity , resulting in insufficient personalization. To address these issues, we propose Fed-PISA, which stands for Fed erated P ersonalized I dentity- S tyle A daptation. To minimize communication costs, Fed-PISA introduces a disentangled Low-Rank Adaptation (LoRA) mechanism: the speaker’s timbre is retained locally through a private ID-LoRA, while only a lightweight style-LoRA is transmitted to the server, thereby minimizing parameter exchange. To harness heterogeneity, our aggregation method, inspired by collaborative filtering, is introduced to create custom models for each client by learning from stylistically similar peers. Experiments show that Fed-PISA improves style expressivity, naturalness, and speaker similarity, outperforming standard federated baselines with minimal communication costs. † † address: \ddagger State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences \dagger Peng Cheng Laboratory \natural University of Chinese Academy of Sciences \flat Institute of Information Engineering, Chinese Academy of Sciences 1 Introduction Modern Text-to-Speech (TTS) systems [ 1 , 2 , 3 ] increasingly leverage voice cloning—the task of synthesizing speech in a target speaker’s voice from arbitrary text inputs— to enable deep personalization: the synthesis of speech that captures not only a target speaker’s vocal timbre but also their unique prosodic patterns and expressive style. The technical pursuit of this nuanced personalization has motivated the emergence of distinct research paradigms. Figure 1: An overview of the FED-PISA framework. On the client side, a private ID-LoRA captures speaker timbre locally, while only a lightweight Style-LoRA is trained and uploaded for aggregation. The server then employs a personalized aggregation strategy to create a custom style model for each client by learning from stylistically similar peers. Among these, two main directions have emerged: high-fidelity zero-shot personalization based on style vectors or large-scale corpus [ 1 , 2 , 4 , 5 ] , along with lightweight on-device deployment via model pruning and Parameter-Efficient Fine-Tuning (PEFT) [ 6 , 7 , 8 , 9 , 10 ] . Despite improvements in speaker similarity and efficiency, these methods share a fundamental limitation: the personalization process is isolated. This isolation creates “style silos”, where a client’s model is constrained by their own limited data, unable to leverage the rich stylistic heterogeneity present across a wider community of peers. Consequently, there is an increasing need for collaborative style learning across different clients under strict preservation of the local data privacy. Under such circumstances, Federated Learning (FL) [ 11 ] offers a privacy-preserving approach that allows multiple clients to collaboratively train a model while keeping their private data on local devices. Existing FL-based TTS frameworks, such as FedSpeech [ 12 ] and Federated Dynamic Transformer [ 13 ] , primarily focus on backbone modification or partitioning to balance speaker-specific personalization with shared global knowledge. Nevertheless, training personalized TTS systems in this manner faces two key challenges. First, these methods often incur substantial computational and communication costs due to complex backbone modifications or large-scale parameter exchanges, hindering their deployment on resource-constrained edge devices. Second, they suppress the severe stylistic heterogeneity exhibited in speech data distribution (e.g., variations in emotion, prosody) to preserve individual speaker timbre. FedSpeech isolates style features locally by parameter masking, while Federated Dynamic Transformer retains personalized layers that capture styles locally, aggregating only the general model. Such suppression limits the model from effectively learn diverse and expressive styles across heterogeneous client data, leading to insufficient model personalization capabilities. This motivates the exploration: Can a federated TTS system effectively learn and utilize heterogeneous styles across clients while maintaining low communication costs? To tackle the aforementioned issues, we propose an efficient and personalized federated TTS framework, namely Fed erated P ersonalized I dentity- S tyle A daptation ( Fed-PISA ). Fed-PISA builds upon the PEFT techniques, specifically inspired by recent studies that utilize Low-Rank Adaptation (LoRA) for single-client TTS personalization [ 8 , 9 , 10 , 14 ] . It adapts the LoRA to a more complex federated setting and consists of two key components: (i) To improve efficiency and preserve speaker timbre, a decoupled LoRA mechanism is introduced. A private ID-LoRA is trained for each client and locally frozen to capture their unique speaker timbre robustly. A lightweight Style-LoRA is used for communication on the server, effectively reducing costs. (ii) To leverage stylistic heterogeneity, we introduce a personalized aggregation strategy. Inspired by collaborative filtering in recommendation systems [ 15 ] , this strategy creates a custom-aggregated style model for each client by prioritizing updates from stylistically similar peers. Our main contributions are threefold 1 1 1 Codes and Audio samples are available at https://huggingface.co/spaces/sDuoluoluos/FedPISA-Demo : • Our work addresses a key challenge overlooked by prior federated TTS frameworks: tackling high communication costs and overcoming the suppression of stylistic heterogeneity . • We propose Fed-PISA , an efficient framework that disentangles speaker timbre from collaborative style learning using LoRA and a novel personalized aggregation mechanism. • Experiments on four public datasets demonstrate that Fed-PISA consistently improves style expressivity, speaker similarity, naturalness, and communication efficiency over federated and nonfederated baselines. 2 Methods 2.1 Backbone and Adapters Let x be the input text and y the target speech waveform. During training, a frozen speech tokenizer E(\cdot) converts y into discrete acoustic/semantic units u=E(y) . We adopt GPT-SoVITS-v4 2 2 2 https://github.com/RVC-Boss/GPT-SoVITS as a two-stage backbone: a GPT-style autoregressive semantic predictor that models p(u\,|\,x,r) conditioned on text x and reference audio r for speaker/style cues and a SoVITS Conditional Flow Matching (CFM) decoder that renders waveform by modeling p(y\,|\,u,r) . LoRA Parameterization. We adopt LoRA parameterized as W=BA , where A\in{R}^{r\times d_{\text{in}}} , B\in{R}^{d_{\text{out}}\times r} . LoRA Placement. ID-LoRA ( W_{\mathrm{ID}} ) is a private, client-specific . It captures speaker timbre and channel coloration, is fine-tuned locally once, and remains permanently frozen—never uploaded or aggregated to the server. In contrast, Style-LoRA ( W_{\mathrm{style}} ) is a federated, globally shared adapter that modulates expressive variation and is updated collaboratively across clients. In the GPT-SoVITS-v4 backbone, the Style-LoRA W_{\mathrm{style}} and the speaker-specific ID-LoRA W_{\mathrm{ID}} are injected into linear projection layers. Within the GPT component, LoRA adapters are applied to all self-attention projections ( q , k , v , out ) as well as the two feed-forward layers (i.e., the up- and down-projection in the MLP block). For the SoVITS component, LoRA is inserted into the attention projections ( q , k , v , out ) within each CFM block. 2.2 Client-Side Optimization Each selected client i loads the frozen backbone, its private ID-LoRA W_{\mathrm{ID}}^{i} , and the current personalized Style-LoRA from the previous round, W_{\mathrm{sty}}^{\prime i,t-1} (or a global model for the first round). Given a local batch (x,y) , we compute acoustic/semantic units u=E(y) . Timbre Cloning. Using neutral speech samples (x,y) from the client, and with a frozen speaker encoder, we update only W_{\mathrm{ID}}^{i} to maximize the cosine similarity between speaker embeddings of the predicted and target waveforms.