Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data

Jiaming Zhang, Huanyi Xie, Meng Ding, Shaopeng Fu, Jinyan Liu, Di Wang · 2026

Read Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data by Jiaming Zhang, Huanyi Xie, Meng Ding, Shaopeng Fu, Jinyan Liu, Di Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03872.txt.

Description

Jiaming Zhang and 5 co-authors ask why private training hurts rare examples most. Their 2026 paper studies DP-SGD, a training method that adds noise and clips gradients so models cannot leak individual data points. Long-tailed data, where many classes appear only a few times, is the test case.

The authors build the first theory of this setting from a feature learning angle. They show that error on the rare subpopulation runs far above average error. Clipping and noise together block the memorization that rare samples need. Experiments on synthetic and real datasets match the theory.

Catalog record

CREATORJiaming Zhang, Huanyi Xie, Meng Ding, Shaopeng Fu, Jinyan Liu, Di Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03872

Browse the full library · About the Tower of Babel