Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models
Qianshan Wei, Jiaqi Li, Zihan You, Yi Zhan, Kecen Li, Jialin Wu, Xinfeng Li, Hengjun Liu, Yi Yu, Bin Cao, Yiwen Xu, Yang Liu, Guilin Qi, [2ex] School of Cyber Science, Engineering, Southeast University, Nanjing, China · 2025
Read Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models by Qianshan Wei, Jiaqi Li, Zihan You, Yi Zhan, Kecen Li, Jialin Wu, Xinfeng Li, Hengjun Liu, Yi Yu, Bin Cao, Yiwen Xu, Yang Liu, Guilin Qi, [2ex] School of Cyber Science, Engineering, Southeast University, Nanjing, China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07077.txt.
Description
Wei and co-authors propose Dual-Priv Pruning, a way to fine-tune multimodal large language models under differential privacy. Privacy noise grows with the size of the model. Large image-and-text models therefore lose a lot of quality when trained this way, and the training is slow and memory-hungry. Their framework prunes in two places at once. It drops visual tokens that carry little new information. It also trims gradient updates during training, based on the size of the noisy updates.
In tests, the method reached results close to non-private training. It used less memory than the standard private training approach. On one common graphics card it used only slightly more memory than a zeroth-order method that performs poorly. On another card it led on memory use. The authors present the work as the first study of private fine-tuning for this class of models.
Catalog record
| CREATOR | Qianshan Wei, Jiaqi Li, Zihan You, Yi Zhan, Kecen Li, Jialin Wu, Xinfeng Li, Hengjun Liu, Yi Yu, Bin Cao, Yiwen Xu, Yang Liu, Guilin Qi, [2ex] School of Cyber Science, Engineering, Southeast University, Nanjing, China |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07077