FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA Subparameter Updates
Sangwoo Park, Seanie Lee, Byungjoo Kim, Sung Ju Hwang · 2025
Read FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA Subparameter Updates by Sangwoo Park, Seanie Lee, Byungjoo Kim, Sung Ju Hwang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07216.txt.
Introduction
Sangwoo Park and co-authors add a privacy shield to federated learning, the scheme where many devices train one model without sending their data to a center. The catch is that each device's model updates still travel to the server, and vision-language models memorize enough that attackers can probe whether a given item was in the training data. Their framework, FedRand, splits a small adapter into halves.
Each device randomly takes some adapter parts from the server and keeps their matching partners private, training both but sending back only the shared parts. In tests, this hid membership far better than rival methods while keeping accuracy close to systems that share everything. The authors present it as a practical way to train shared models on sensitive images and text.
Catalog record
| CREATOR | Sangwoo Park, Seanie Lee, Byungjoo Kim, Sung Ju Hwang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07216
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA Subparameter Updates Sangwoo Park, Seanie Lee, Byungjoo Kim, Sung Ju Hwang Abstract Federated Learning (FL) is a widely used framework for training models in a decentralized manner, ensuring that the central server does not have direct access to data from local clients. However, this approach may still fail to fully preserve data privacy, as models from local clients are exposed to the central server during the aggregation process. This issue becomes even more critical when training vision-language models (VLMs) with FL, as VLMs can easily memorize training data instances, making them vulnerable to membership inference attacks (MIAs). To address this challenge, we propose the FedRand framework, which avoids disclosing the full set of client parameters. In this framework, each client randomly selects subparameters of Low-Rank Adaptation (LoRA) from the server and keeps the remaining counterparts of the LoRA weights as private parameters. After training both parameters on the client’s private dataset, only the non-private client parameters are sent back to the server for aggregation. This approach mitigates the risk of exposing client-side VLM parameters, thereby enhancing data privacy. We empirically validate that FedRand improves robustness against MIAs compared to relevant baselines while achieving accuracy comparable to methods that communicate full LoRA parameters across several benchmark datasets. 1 Introduction Vision-language models (VLMs) ( Alayrac et al. 2022 ; Zhu et al. 2023 ; Liu et al. 2023 ) have demonstrated remarkable performance in various multi-modal tasks, such as visual question answering ( Dai et al. 2023 ; Liu et al. 2023 ) and image captioning ( Li et al. 2023 ) . However, deploying VLMs in real-world scenarios raises significant concerns about data privacy. These models can easily memorize training data ( Carlini et al. 2021 ; Carlini et al. 2023 ) , including sensitive information such as private photographs or medical diagnosis records. Adversarial attackers can exploit this vulnerability to perform a membership inference attack ( Shokri et al. 2017 ) , which aims to detect whether a specific data instance is part of the training dataset. Federated learning ( McMahan et al. 2017 , FL;) is a distributed learning framework in which each local client receives global parameters from a central server, trains a local model on its private dataset, and periodically sends the local model back to the server for aggregation. It offers a potential solution to address privacy concerns, as the central server cannot directly access the private dataset. However, naively transmitting local model parameters back to the central server remains vulnerable to membership inference attacks, as attackers can potentially reconstruct the local client model by intercepting its parameters during the aggregation stage. This issue is particularly critical when fine-tuning vision-language models (VLMs), as their large capacity to memorize private training data amplifies the privacy risks. Figure 1: (a) . At each round r , each local client selects a LoRA weight either A_{r} or B_{r} for initialization from the server and initializes the other counterparts of LoRA weights using the previous round’s client model parameters as private parameters. (b) . After updating both parameters, only the non-private parameters are sent back to the server and aggregated to update the LoRA weights of the central server. To address the privacy issue, we propose a simple yet privacy-enhanced federated learning (FL) framework, dubbed FedRand . In this framework, clients randomly select a subset of parameters provided by the server and keep the remaining parameters as client-specific private ones. After updating both the selected parameters and their client-specific private parameters, only the non-private parameters are transmitted back to the server for the model update. Specifically, we first apply Low-Rank Adaptation ( Hu et al. 2022 , LoRA;) matrices A and B to the pre-trained weight W_{0} of a VLM. The pre-trained weight W_{0} is fixed and shared across all clients and the server. At each round of updates, each local client model receives the LoRA weights A and B from the server. Each client then randomly selects either A or B and initializes the counterpart of the LoRA weights using the parameters from the previous round as client-specific private ones( Figure 1 a ). After updating both parameters on the client’s private training dataset, the client-specific parameters remain hidden, and only the remaining parameters are sent back to the server. Finally, the parameters A and B from the clients are averaged to form the new LoRA weights of the server model ( Figure 1 b ). Since the client-specific private parameters are kept hidden, adversarial attackers cannot fully reconstruct the client model parameters by intercepting the parameters transmitted to the server. This design makes FedRand more robust against membership inference attacks. Furthermore, sending only non-private parameters significantly reduces the communication cost between the server and clients compared to the model that communicates all LoRA weights between the server and clients. Figure 2: Trade-off between task performance (CIDEr) and vulnerability to membership inference attacks (AUROC of MIA) on MSCOCO dataset. We empirically validate our proposed FedRand on visual question answering and image captioning tasks using the ScienceQA ( Lu et al. 2022 ) , MSCOCO ( Lin et al. 2014 ) , and NoCaps ( Agrawal et al. 2019 ) datasets. Experimental results demonstrate that FedRand significantly improves the trade-off between accuracy and robustness against membership inference attacks ( Figure 2 ) while reducing communication costs between the server and clients compared to other relevant baselines. Our contributions and findings are summarized below: • We show that even fine-tuning VLMs with FL remains vulnerable to membership inference attacks due to the exposure of client model parameters, posing significant privacy concerns. • To address these privacy concerns, we propose FedRand. First, a client randomly selects subparameters of LoRA weights from the server and updates both the selected parameters and client-specific private parameters. Only the non-private parameters are sent back to the server, preventing the exposure of the full local model parameters. • We experimentally demonstrate that FedRand enhances robustness against membership inference attacks while achieving performance comparable to models that communicate full LoRA weights between the server and clients. 2 Related Work Federated learning. Federated Learning (FL) is a decentralized machine learning approach that allows multiple clients to collaboratively train a shared model without sharing their private data, thereby preserving privacy and security. FedAvg ( McMahan et al. 2017 ) , one of the most widely used algorithms in FL, updates a global model by averaging the model parameters trained on each client’s private dataset. While many variants of FedAvg have been proposed ( Li et al. 2020 ; Yu et al. 2020 ; Acar et al. 2021 ; Zhang et al. 2024 ) , they remain vulnerable to membership inference attacks because clients’ parameters are exposed to the server. In another line of work, methods like FedPer ( Arivazhagan et al. 2019 ) and FedPara ( Hyeon-Woo et al. 2022 ) distinguish client-specific private parameters from global parameters shared between the server and clients to reduce communication costs. However, although these methods avoid exposing client parameters to the server, they fail to strike a balance between accuracy and robustness against membership inference attacks. Membership inference attack. Although FL avoids sharing private