Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models
Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova · 2025
Read Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models by Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13904.txt.
Introduction
Tran, Sun, Patterson, and Milanova design a way to personalize multimodal AI helpers across many users without pooling their private data. In federated prompt learning, each site tunes small prompt pieces locally on a shared model. Their 2025 method must balance three pulls: fitting each user, working for all users, and keeping data private with added noise.
Their scheme splits each local prompt into a shared low-rank part that carries general skill and a leftover part that carries personal flavor. Privacy noise is applied at two scales, local for the split parts and global for the shared prompt. Tests against rival methods show the balance holds up better. The authors aim the work at support and operations tools that mix text, image, and sound.
Catalog record
| CREATOR | Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13904
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova Abstract Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks. 1 Introduction In recent years, there has been rapid advancement in multimodal large language models (LLMs) that integrate multiple modality information, including text, images, audio, and video, to enhance the comprehension and generation capabilities. Vision-Language Models (VLMs) such as CLIP ( Radford et al., 2021 ) are a variant of multimodal LLMs that learn transferable image and text representations, making them highly effective in applications such as image captioning and visual search. One proposed setting for deploying VLMs is a Federated Learning framework that allows multiple organizations or clients to collaboratively train a global model without directly sharing their local training data. However, fine-tuning pre-trained VLMs in a FL system is time-consuming and resource-intensive given the massive number of parameters each VLM has. This gives rise to Federated Prompt Learning (FPL) which only fine-tunes the soft prompt embedding while freezing the rest of the VLM model parameters in the FL system ( Guo et al., 2023b ; Cui et al., 2024 ) . In FPL, each client fine-tunes their customized prompt using their local data and shares the prompt with a central server for generalization purposes. The clients can distribute their fine-tuned prompts as prompt providers to public users who wish to perform downstream inference tasks, also known as the prompt as a service (PaaS) paradigm ( Wu et al., 2024 ; Yao et al., 2024 ; Huang et al., 2023 ) . One significant challenge in such distributed systems is the presence of data heterogeneity, i.e., organizations often have non-identical and non-independent (non-IID) data distributions, which can vary widely due to factors such as demographics, usage patterns, or device capabilities. To address this, personalized FL has emerged to tailor models to the unique data characteristics of each client rather than solely improving a global model. Personalized FL focuses on learning customized models for each client, reflecting the heterogeneity of their data ( He et al., 2020 ; Dinh et al., 2020 ) . In the context of personalized FPL, the goal is for each client to learn and utilize personalized prompts that better align with their specific data and application needs. Nevertheless, over-personalization can lead to local data overfitting, preventing the model to generalize well on non-training data. Clients in an FPL framework may opt to distribute their customized prompts post training to public users for downstream tasks. However, these users may have different types of inputs that the client’s prompt is not well generalized to, resulting in suboptimal performance. Consequently, it is crucial to achieve a nuanced balance between personalization and generalization in a heterogeneous FPL system. In addition to balancing the tradeoff between personalization and generalization, privacy poses another critical concern in FPL, especially in sensitive domains such as finance, law, and healthcare. In the PaaS framework, the distributed trained prompts are shown to be susceptible to Membership Inference Attack (MIA), potentially exposing details about individual clients’ training data ( Wu et al., 2024 ) . To address this issue, one may consider Differential Privacy (DP) ( Dwork et al., 2014 ) which ensures an adversary cannot reliably detect the presence or absence of a data sample based on the output information. However, balancing personalization and privacy under data heterogeneity is a challenging task. The non-IID nature of the data allows clients to better learn their personalized prompt, but it amplifies the performance degradation caused by DP due to the high data sensitivity, impairing both personalization and generalization capabilities. Thus, the key question we aim to address is: How can we effectively balance personalization, generalization, and privacy in a data heterogeneous FPL system? To tackle the above question, we proposed a Differentially Private Federated Prompt Learning (DP-FPL) approach that leverages low-rank factorization and DP as part of the prompt learning process. In our framework, each client simultaneously learns a global prompt and a local prompt. The global prompt is shared in a FL manner for generalized knowledge transfer, while the local prompt is retained at each client site for personalization. Our contributions are threefold. • We propose a privacy-preserving personalized federated prompt learning approach with Differential Privacy for multimodal LLMs. We factorize the local prompt into two lower rank components with an additional residual term. The factorized low-rank components allow the model to capture broader patterns that are beneficial across different data distributions, aiding the generalization capability of each client. The residual term is crucial for retaining the expressiveness lost during the factorization process, thereby preserving the client-specific learning and improving personalization. • We preserve privacy by utilizing both Global Differential Privacy (GDP) and Local Differential Privacy (LDP). Unlike conventional methods that apply noise uniformly to the entire prompt, we judiciously apply LDP to the two low-rank components of the local prompt, and GDP to the global prompt. Our privacy mechanism mitigates the effect of DP noise on model performance while preserving the privacy guarantee post training. • We conduct extensive experiments on widely adopted datasets to evaluate our proposed method against other benchmarks. The experimental results demonstrate superior performance of our proposed method in balancing personalization and generalization while mitigating the model degradation caused by DP noise. 2 Related Work Personalized Federated Learning. There are several existing approaches that aim to learn personalized models for clients in FL settings, including clustering ( Ghosh et al., 2020 ; Berlo et al., 2020 ; Shahid et al., 2021 ) , regularization ( Shoham et al., 2019 ; Dinh et al., 2020 ; Li et al., 2020 ) and knowledge distillation ( Li & Wang, 2019 ; He et al., 2020 ; Fang & Ye, 2022 ) . Personalized FL is most commonly approached as a multi-task learning problem that simultaneously learns two models for each client: a global model for generalized knowledge and a local model for personalized data. Existing methods accomplish this by decoupling the model parameters or layers into global and local learning components ( Arivazhagan et al., 2019 ; Deng et al., 2020 ; Zhang et al., 2020 ; Collins et al., 2021