DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
Haaris Mehmood, Jie Xu, Karthikeyan Saravanan, Rogier Van Dalen, Mete Ozay · 2026
Read DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models by Haaris Mehmood, Jie Xu, Karthikeyan Saravanan, Rogier Van Dalen, Mete Ozay (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10272.txt.
Introduction
Haaris Mehmood and four colleagues protect privacy in collaborative AI training. Federated learning lets devices train a shared language model while keeping user data on the device, yet the updates devices send can still leak sensitive information. Their 2026 paper introduces DP-LAC, a method for differentially private federated fine-tuning that guards those updates.
Private training clips each client update to a threshold and adds noise scaled to that threshold. Existing adaptive methods tune the threshold but demand fiddly hyperparameters that can eat into the privacy budget. DP-LAC first estimates a starting threshold close to the best value, using private histogram estimation. It then adapts that threshold during training without spending extra privacy budget or adding new hyperparameters. It beat leading adaptive methods and vanilla private training, gaining 6.6 percent average accuracy.
Catalog record
| CREATOR | Haaris Mehmood, Jie Xu, Karthikeyan Saravanan, Rogier Van Dalen, Mete Ozay |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10272
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models Haaris Mehmood, Jie Xu, Karthikeyan Saravanan, Rogier Van Dalen, Mete Ozay Abstract Federated learning (FL) enables the collaborative training of large - scale language models (LLMs) across edge devices while keeping user data on - device. However, FL still exposes sensitive information through client - provided gradients. Differentially private stochastic gradient descent (DP - SGD) mitigates this risk by clipping each client’s contribution to a threshold C C and adding noise proportional to C C . Existing adaptive clipping techniques dynamically adjust C C but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP - LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP - LAC outperforms both state - of - the - art adaptive clipping methods and vanilla DP - SGD, achieving an average accuracy gain of 6.6%. † † address: Samsung R&D Institute UK (SRUK), Samsung AI Centre Cambridge [email protected] ©2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. 1 Introduction Large language models (LLMs) now dominate a wide array of NLP tasks [ 24 , 28 ] . When several stakeholders wish to train such models collaboratively while keeping user data local, federated learning (FL) offers a natural solution: each client trains on its own device and only model updates (pseudo - gradients) are exchanged with a central server [ 20 , 16 , 10 ] . Nonetheless, these updates can leak sensitive information, especially for memorizing LLMs [ 13 , 33 , 9 , 17 ] . Differential privacy (DP) [ 12 ] supplies a formal guarantee that protects client contributions against reconstruction attacks [ 14 , 31 ] . In DP, a randomized mechanism \mathcal{M}:\mathcal{D}\rightarrow\mathcal{R} satisfies (\varepsilon,\delta) - DP if for every pair of adjacent datasets \mathcal{D},\mathcal{D}^{\prime} and every output set \mathcal{C}\subseteq\mathcal{R} : \Pr\!\bigl[\mathcal{M}(\mathcal{D})\in\mathcal{C}\bigr]\leq e^{\varepsilon}\Pr\!\bigl[\mathcal{M}(\mathcal{D}^{\prime})\in\mathcal{C}\bigr]+\delta\vskip-1.49994pt (1) where \varepsilon is the privacy budget and \delta the failure probability [ 12 ] . DP - FedAvg [ 21 ] implements DP in FL by applying the Gaussian mechanism to each client’s post-training weight difference (pseudo-gradient). The procedure is two - step: 1. Each client clips its pseudo-gradient \Delta , to a fixed \ell_{2} -norm threshold C : \tilde{\Delta}_{k}=\Delta_{k}\cdot\min(1,C/\lVert\Delta_{k}\rVert_{2}) 2. The clipped update \tilde{\Delta}_{k} is perturbed with Gaussian noise \mathcal{N}(0,\sigma^{2}I) , where the standard deviation \sigma=C\cdot z is determined by the noise multiplier z that encodes the client’s privacy parameters (\varepsilon,\delta) and participation frequency. The clipping threshold C is pivotal: if C is too large, the added noise dominates the signal; if it is too small, legitimate gradient directions are severely distorted, introducing bias. Consequently, selecting C involves a delicate bias–variance trade - off [ 11 , 8 , 35 ] . In practice, researchers set C to a percentile (e.g., the 95th) of the distribution of client gradient norms [ 3 ] , yet the optimal percentile is highly sensitive to model architecture, dataset, and other hyperparameters. These observations expose three main limitations of current approaches: (i) manual tuning is time - consuming and privacy - costly because each tuning attempt consumes part of the privacy budget; (ii) online clipping schedules introduce additional hyperparameters (e.g., learning rates for the clipping schedule) that must be calibrated, further complicating deployment; and (iii) a fixed C set at training start yields sub - optimal robustness, as any shift in data distribution or model dynamics can render the chosen threshold ineffective. Towards this, in this paper, we present DP - LAC (Differentially Private Federated Fine - tuning with Lightweight Adaptive Clipping), a method that automatically adapts the clipping threshold C during LLM fine - tuning under DP - FL, without adding any extra hyperparameters. In a realistic setup with 1,000 clients and accounting for the privacy cost of hyperparameter tuning, DP - LAC outperforms existing adaptive clipping methods by an average of 6.6% with 5-15x faster hyperparameter grid-search times [ 11 , 8 ] . Figure 1 presents an overview of our method. Figure 1 : Overview of our adaptive clipping method, DP-LAC. u_{i} is a one-hot encoding vector and \Delta_{i} the post-training weight difference. v_{t} is the server-side validation loss and C_{t} the clipping threshold at round t . More details of our method can be found in Algorithm 1 . 2 Background 2.1 Differentially Private Federated Learning (DP-FL) Federated learning trains a model from data held by distributed clients while keeping the raw data on each device. In each communication round t , the server broadcasts the current global model \bm{W}_{t} to a randomly selected subset of clients \mathcal{K}_{t} . Every client k\in\mathcal{K}_{t} performs local training, yielding a locally updated model \bm{W}_{k} , and sends the model update \Delta_{k}=\bm{W}_{k}-\bm{W}_{t} back to the server. The server then aggregates these updates to produce the global model for the next round. Differential privacy can protect FL at the sample level or at the user level. In this paper we focus on user - level DP, introduced by [ 21 ] . To obtain (\varepsilon,\delta) -user - level DP, the \ell_{2} -norm of each client’s update \Delta_{k} is clipped to a threshold C and then a Gaussian noise vector \mathcal{N}(0,\sigma^{2}\mathbf{I}) is added to the sum of updates, where \sigma^{2}=C^{2}z^{2} and z is the noise multiplier. This two–stage process is the core of DP - FedAvg, the defacto algorithm for DP-FL [ 21 ] . 2.2 Adaptive Clipping for DP-FL Previous adaptive - clipping methods [ 3 , 11 , 8 , 26 ] dynamically adjust the clipping threshold to improve DP optimization. However, they introduce additional hyperparameters that must be tuned, incurring significant computational effort and consuming part of the privacy budget [ 18 , 25 ] . In [ 18 ] (Theorem 3.5), an algorithm that tunes its hyperparameters achieves at most (3\varepsilon,0) -DP if the underlying algorithm satisfies (\varepsilon,0) -DP. A concurrent study [ 7 ] proposes an automatic learning - rate selection by fitting a quadratic private - loss function every K rounds; this approach, however, requires privatising three losses per cycle, introduces a hidden hyperparameter \gamma , and is designed for central DP - SGD rather than DP - FL. 3 Methodology Algorithm 1 DP-LAC 1: function ServerLoop 2: parameters: T (num. rounds), \mathcal{K} (clients), q\in(0,1] (sampling rate), \varepsilon (privacy budget), \delta (failure prob.), E (local epochs), \beta (batch size), \eta (local learning rate), F (loss function), \bm{s}\in\mathbb{R}^{d} ( d clip. thresholds), \bm{m}\in\mathbb{R}^{e} ( e multiplier choices), \bm{W}_{0} (pretrained model weights), \mathcal{D}_{val} (server validation set) 3: z\leftarrow GetNoiseMult (T,q,\varepsilon,\delta) \triangleright Noise multiplier 4: v_{0}\leftarrow F(\bm{W}_{0},\mathcal{D}_{val}) \triangleright Server validation loss 5: for t\leftarrow 1,T do 6: Sample \mathcal{K}_{t}\subseteq\mathcal{K} uniformly at random (probability q ) 7: if t=1 then 8: C_{1}\leftarrow InitC (\mathcal{K}_{t},\bm{W}_{0}) 9: \bm{W}_{1},v_{1}\leftarrow\bm{W}_{0},v_{0} 10: else 11: