Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data
Jiaming Zhang, Huanyi Xie, Meng Ding, Shaopeng Fu, Jinyan Liu, Di Wang · 2026
Read Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data by Jiaming Zhang, Huanyi Xie, Meng Ding, Shaopeng Fu, Jinyan Liu, Di Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03872.txt.
Description
Jiaming Zhang and 5 co-authors ask why private training hurts rare examples most. Their 2026 paper studies DP-SGD, a training method that adds noise and clips gradients so models cannot leak individual data points. Long-tailed data, where many classes appear only a few times, is the test case.
The authors build the first theory of this setting from a feature learning angle. They show that error on the rare subpopulation runs far above average error. Clipping and noise together block the memorization that rare samples need. Experiments on synthetic and real datasets match the theory.
Catalog record
| CREATOR | Jiaming Zhang, Huanyi Xie, Meng Ding, Shaopeng Fu, Jinyan Liu, Di Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-03872