Scaling Laws for Differentially Private Language Models

Ryan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A. Choquette-Choo, Badih Ghazi, George Kaissis, Ravi Kumar, Ruibo Liu, Da Yu, Chiyuan Zhang · 2025

Read Scaling Laws for Differentially Private Language Models by Ryan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A. Choquette-Choo, Badih Ghazi, George Kaissis, Ravi Kumar, Ruibo Liu, Da Yu, Chiyuan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18914.txt.

Introduction

Rules of thumb called scaling laws let labs predict how much a language model improves as training grows, saving costly trial and error. Ryan McKenna and co-authors extend those laws in a 2025 paper to training with differential privacy, the noise-adding method that protects people whose text is in the data. Private training behaves differently, so the old laws mislead.

Their new laws capture the three-way trade among computing power, privacy strength, and model quality, and they point to the best training settings in many situations. The authors say the results give practical guidance for building strong models on sensitive data without wasting compute or weakening privacy.

Catalog record

CREATORRyan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A. Choquette-Choo, Badih Ghazi, George Kaissis, Ravi Kumar, Ruibo Liu, Da Yu, Chiyuan Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18914

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling Laws for Differentially Private Language Models Ryan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A. Choquette-Choo, Badih Ghazi, George Kaissis, Ravi Kumar, Ruibo Liu, Da Yu, Chiyuan Zhang Abstract Scaling laws have emerged as important components of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper-parameter choices that would otherwise be expensive. LLMs also rely on large, high-quality training datasets, like those sourced from (sometimes sensitive) user data. Training models on this sensitive user data requires careful privacy protections like differential privacy (DP). However, the dynamics of DP training are significantly different, and consequently their scaling laws are not yet fully understood. In this work, we establish scaling laws that accurately model the intricacies of DP LLM training, providing a complete picture of the compute-privacy-utility tradeoffs and the optimal training configurations in many settings. 1 Introduction Large language models (LLMs) are revolutionizing how we interact with technology, powering everything from instant translations and concise summaries to complex reasoning and creative content generation ( Achiam et al., 2023 ; Gemini Team, 2023 ) . Training increasingly large models over increasingly large datasets has been a key driver of success for these LLMs, with frontier models being trained for millions of GPU-hours ( Anil et al., 2023 ) and increasingly many trillions of tokens ( Gemma Team et al., 2024a ; Gemma Team et al., 2024b ) . Scaling laws for neural language models have been crucial because they provide a framework for understanding and predicting the performance gains achievable with increased computational resources, and importantly, guide the optimal allocation of that compute budget between model size and dataset size ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ) . The scale of data driving LLM progress also creates a critical privacy challenge. State-of-the-art models train on massive, diverse datasets ( Dubey et al., 2024 ; Gemma Team et al., 2024a ) that are also distributed ( Carlini et al., 2024 ) making it difficult to exclude inadvertently shared personal information. Paradoxically, user data, a key privacy concern, is also crucial for advancing LLM capabilities. User interactions provide invaluable feedback for generating realistic synthetic data ( Afonja et al., 2024 ; Kurakin & Ponomareva, 2024 ) and aligning models with human values ( Stiennon et al., 2020 ) , reflecting real-world use cases better than web-scraped text. However, direct training on sensitive user data is risky due to memorization and regurgitation ( Carlini et al., 2021 ; Carlini et al., 2023 ; Ippolito et al., 2022 ; Lukas et al., 2023 ; Biderman et al., 2023 ; Prashanth et al., 2024 ) . This tension—the need for user data versus protecting user privacy—is addressed by differential privacy (DP) ( Dwork et al., 2006 ) . While DP offers a principled solution to the tension between data utility and privacy in LLM training, applying it in practice, especially to large-scale models, presents significant challenges. DP mechanisms like DP-SGD ( Abadi et al., 2016 ) and its variants introduce computational overhead, implementation complexity ( Subramani et al., 2021 ) , and utility degradation ( Bassily et al., 2014 ) . While it is generally well-known that DP-SGD benefits substantially from training with very large batch sizes ( Anil et al., 2022 ; De et al., 2022 ; Ponomareva et al., 2023 ) , little work has been done to understand the conditions under which this holds in compute-constrained settings, i.e., when an increase in batch size must be coupled with a decrease in model size or the number of iterations. In part due to this reliance on large batch sizes, the largest models trained with DP today have hundreds of millions, rather than billions, of parameters ( Anil et al., 2022 ; Li et al., 2022 ; Berrada et al., 2023 ; Ghalebikesabi et al., 2023 ; Charles et al., 2024 ) . To train large models with DP, it is crucial to spend both the compute budget and the privacy budget judiciously. In this work, we pave the way towards training at the billion-parameter scale by initiating a study on the scaling laws of DP training . To that end, we extend traditional scaling laws to consider a compute-privacy-utility tradeoff, accounting for intricacies and additional variables introduced by DP training. Through a rigorous set of experiments, we empirically model this trade-off, and provide a thorough analysis of these experimental results to answer a number of scaling law-style questions, finding (among other things) that: • The compute budget allocation predicted by non-private scaling laws is far from optimal under DP, even for huge privacy budgets, confirming the need for our study. • However, we can accurately predict the optimal breakdown of the compute budget into model size, batch size, and iterations for virtually any privacy budget and dataset size. These compute-efficient training configurations save 5\times to 100\times compute compared to baseline configurations, while retaining comparable privacy and utility. • The optimal model size is typically at least an order of magnitude smaller with DP than without. This provides insight into the challenges of training large billion-parameter or larger language models with DP. • In the DP setting, increasing the compute budget can sometimes yield little to no reduction in the loss unless accompanied by a corresponding increase in the privacy budget or dataset size. 2 Preliminaries and Problem Setup Key Definition \epsilon Privacy budget N Data budget C Compute budget B Batch size T Iterations S Sequence length M Model parameters \bar{\sigma} noise-batch ratio Our dataset \mathcal{D} consists of text sequences, where each individual contributes a single sequence \textbf{x}=(x_{1},\ldots,x_{S}) of S tokens, and each token is drawn from a predefined vocabulary \mathcal{V} . We let N denote the total number of individuals contributing to the dataset. Masked Language Modeling. In this work we focus on the masked language modeling task ( Devlin et al., 2019 ) , where each sequence has a chosen fraction p_{\mathrm{mask}} of tokens masked out, i.e., replaced with a special masking token [MASK], uniformly at random. The goal is to predict the original token for each masked token using the entire context (bidirectionally). Let \bar{\textbf{x}} represent the original sequence of tokens but masked using the above procedure and \mathcal{M} the ids of the masked tokens in \bar{\textbf{x}} . For a given parameter vector \theta\in\mathbb{R}^{M} , the language model defines a conditional probability p_{\theta}(x_{j}~\mid~\bar{\textbf{x}}) for each j\in\mathcal{M} , and the goal is to find \theta to maximize the likelihood of all masked training tokens. Differential Privacy. A randomized mechanism \mathcal{A} satisfies (\epsilon,\delta) -DP ( Dwork et al., 2006 ) if, for any two datasets \mathcal{D} , \mathcal{D}^{\prime} that differ by a single individual, all subsets \mathcal{O} of possible outputs of \mathcal{A} and \epsilon>0,0\leq\delta<1 : \Pr[\mathcal{A}(\mathcal{D})\in\mathcal{O}]\leq e^{\epsilon}\Pr[\mathcal{A}(\mathcal{D}^{\prime})\in\mathcal{O}]+\delta. (1) DP-SGD. DP-SGD is a widely used algorithm to train neural networks with DP. It attains provable DP guarantees through limiting the contribution ( sensitivity ) of each example by clipping its gradient to some \ell_{2} -norm (wlog, 1 ), and then adding isotropic Gaussian noise to the averaged clipped gradients; see Algorithm 1 for pseudo-code. Our algorithm is a slight generalization of the original DP-SGD ( Abadi et

Read the complete text