Differentially Private Subspace Fine-Tuning for Large Language Models

Lele Zheng, Xiang Wang, Tao Zhang, Yang Cao, Ke Cheng, Yulong Shen · 2026

Read Differentially Private Subspace Fine-Tuning for Large Language Models by Lele Zheng, Xiang Wang, Tao Zhang, Yang Cao, Ke Cheng, Yulong Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11113.txt.

Introduction

Lele Zheng and co-authors fine-tune large language models in a subspace under differential privacy. Fine-tuning adapts a model to a task, often on sensitive data. Differential privacy gives strict guarantees, but adding noise across the full parameter space creates huge perturbations. Training wobbles and quality drops. The authors confine the noise to a subspace instead.

Working in a lower-dimensional subspace shrinks the noise's total size. Privacy guarantees can hold while the model still learns. The balance matters for hospitals, law firms, and banks that fine-tune on private records. The work offers a steadier path to private adaptation than naive noise over billions of parameters.

Catalog record

CREATORLele Zheng, Xiang Wang, Tao Zhang, Yang Cao, Ke Cheng, Yulong Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Differentially Private Subspace Fine-Tuning for Large Language Models Lele Zheng, Xiang Wang, Tao Zhang, Yang Cao, Ke Cheng, Yulong Shen Abstract Fine-tuning large language models on downstream tasks is crucial for realizing their cross-domain potential but often relies on sensitive data, raising privacy concerns. Differential privacy (DP) offers rigorous privacy guarantees and has been widely adopted in fine-tuning; however, naively injecting noise across the high-dimensional parameter space creates perturbations with large norms, degrading performance and destabilizing training. To address this issue, we propose DP-SFT, a two-stage subspace fine-tuning method that substantially reduces noise magnitude while preserving formal DP guarantees. Our intuition is that, during fine-tuning, significant parameter updates lie within a low-dimensional, task-specific subspace, while other directions change minimally. Hence, we only inject DP noise into this subspace to protect privacy without perturbing irrelevant parameters. In phase one, we identify the subspace by analyzing principal gradient directions to capture task-specific update signals. In phase two, we project full gradients onto this subspace, add DP noise, and map the perturbed gradients back to the original parameter space for model updates, markedly lowering noise impact. Experiments on multiple datasets demonstrate that DP-SFT enhances accuracy and stability under rigorous DP constraints, accelerates convergence, and achieves substantial gains over DP fine-tuning baselines. Code — https://github.com/XidianNss/DP-SFT Introduction Large language models (LLMs), such as BERT and GPT, have achieved remarkable success in natural language processing and generation tasks ( 8 ; 35 ; 2 ) . To optimize their performance on specific downstream tasks, fine-tuning is essential ( 27 ; 13 ; 9 ) . As shown in Fig. 1 , this process allows pre-trained models to be adapted to task-specific data, enhancing accuracy and effectiveness. However, fine-tuning often relies on datasets that contain sensitive information, such as personal identities, financial data, or private conversations. The potential leakage of such information poses serious threats to user privacy and security ( 7 ) . Therefore, balancing privacy protection with the performance of fine-tuning has become a critical challenge. Figure 1: Fine-tuning Process of Large Language Models Differential privacy (DP) ( 10 ) provides strict privacy guarantees for sensitive data. The core principle of DP is to ensure that an algorithm’s output remains consistent despite small changes in a single data sample. In other words, minor adjustments to an individual’s data should not significantly alter the algorithm’s results, thus ensuring the protection of user privacy. DP achieves this by introducing noise into the model, making it impossible for external observers to infer any individual data from the model’s output. Recently, DP has become widely applied to large model training and fine-tuning, particularly in applications involving sensitive data. ( 1 ) introduced DPSGD, which incorporates Gaussian noise into gradients during deep learning, laying the theoretical foundation for privacy protection in the fine-tuning of large language models. Building on this, ( 17 ) proposed AnaDP, a method that dynamically allocates noise and privacy budgets based on the importance of model parameters during specific training steps. Their research demonstrated that AnaDP performs exceptionally well across multiple datasets, narrowing the performance gap between DP fine-tuning and standard fine-tuning while maintaining privacy. Despite significant progress in the application of differential privacy to fine-tuning, existing methods still face the challenge of the curse of dimensionality. To ensure privacy protection, noise must typically be injected into the entire high-dimensional parameter space. This approach increases the noise magnitude substantially, leading to unnecessary performance degradation and training instability, which ultimately affects the model’s effectiveness. For instance, given a fixed privacy parameter, the variance of the Gaussian noise added to the gradients increases linearly with the parameter dimensionality. The introduction of high-dimensional noise presents two major challenges. First, the randomness of the noise significantly disturbs each gradient update direction, making the training process more unpredictable and reducing the model’s final performance. Second, the added noise negatively impacts the stability of training results. In the absence of noise, the model typically converges gradually over several iterations. However, with the introduction of high-dimensional noise, the training results often exhibit large fluctuations, making it difficult for the model to converge quickly. Therefore, achieving differential privacy protection in large language model fine-tuning without sacrificing model performance remains a significant challenge. To address the challenges mentioned above, we draw inspiration from subspace differential privacy and propose a DP-SFT mechanism based on gradient subspace optimization. Our approach is founded on the observation that, despite the vast number of parameters in large language models, effective gradient updates during fine-tuning rely heavily on a few dominant gradient directions. These directions form a compact low-dimensional subspace that is significantly smaller than the original parameter space. Consequently, our method first constructs a gradient subspace by extracting the trajectory of full parameter fine-tuning. It then projects the training gradients onto this subspace, adds noise to meet the differential privacy requirements, and finally projects the noisy subspace back into the high-dimensional space to complete the gradient update. Intuitively, DP-SFT reduces the required noise by significantly lowering the dimensionality of the subspace (by several orders of magnitude). Specifically, our contributions are as follows: 1. We propose Differentially Private Subspace Fine-Tuning (DP-SFT), a novel two-stage framework that injects privacy noise only into a task-specific low-dimensional subspace of gradients. This approach significantly reduces noise magnitude while preserving formal DP guarantees, overcoming the dimensionality challenge in DP optimization for LLMs. 2. We demonstrate subspace transferability across tasks, enabling the subspace to be constructed on public or related datasets without consuming the privacy budget. Our analysis and experiments show that the subspace captures general task geometry rather than private sample-specific information, making DP-SFT practical and privacy-efficient. 3. Extensive experiments demonstrate that DP-SFT achieves near-non-private performance and consistently outperforms strong baselines under different privacy budgets. It sets a new state-of-the-art in privacy-utility trade-offs for differentially private fine-tuning of LLMs. Related Works A straightforward approach to ensuring differential privacy during fine-tuning is to inject noise into gradients, as demonstrated in DP-SGD ( 1 ) and DP-Adam ( 16 ) . However, due to the curse of dimensionality, directly perturbing gradients in large language models (LLMs) often leads to significant accuracy degradation. To alleviate this issue, ( 32 ) proposes a reparameterized gradient perturbation, which reparameterizes each high-rank weight matrix into two low-rank matrices along with a residual matrix. Noise is then added only to the gradients of the low-rank components, and the perturbed gradients are projected back to update the original high-rank weights. However, applying such reparameterization at every update step can introduce training instability. More recent work ( 30 ) leverages parameter-efficient fine-tuning techniques, such as LoRA ( 13 ) , Adapter ( 12 ) , and Compacter ( 15 ) , to reduce the number of perturbed parameters. These methods inject noise only into the gradients of additional lightweight plug-in modules, rather than the full model. While this improves efficiency, ( 18 ) argues that full-parameter fine-tuning remains essential in many domains. They introduce ghost clipping, a reparameterization-based technique that avoids explicitly instantiating per-example gradients. Nevertheless, existing low-rank reparameterization methods ( 5 ; 34 ) still require a large number of retained dimensions to maintain accuracy. For example, even after reparameterization, fine-tuning LLaMA-7B still involves updating hundreds of millions of parameters ( 5 ) .

Read the complete text