Membership and Memorization in LLM Knowledge Distillation

Ziqi Zhang, Ali Shahin Shamsabadi, Hanxiao Lu, Yifeng Cai, Hamed Haddadi · 2025

Read Membership and Memorization in LLM Knowledge Distillation by Ziqi Zhang, Ali Shahin Shamsabadi, Hanxiao Lu, Yifeng Cai, Hamed Haddadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07054.txt.

Introduction

Ziqi Zhang and four co-authors study privacy risks when a large language model teaches a smaller one in a 2025 paper. Distillation shrinks the cost of running big models, but the small student may inherit private facts the teacher saw in training. The team test six distillation methods across tasks, teacher families, and student sizes.

All methods carried some risk that outsiders could tell whether a given text was in the teacher's training data, or extract memorized text. Risk varied by method, and two kinds of privacy risk did not move together. Risk also varied sharply across parts of the model, the analysis shows.

Catalog record

CREATORZiqi Zhang, Ali Shahin Shamsabadi, Hanxiao Lu, Yifeng Cai, Hamed Haddadi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07054

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Membership and Memorization in LLM Knowledge Distillation Ziqi Zhang, Ali Shahin Shamsabadi, Hanxiao Lu, Yifeng Cai, Hamed Haddadi Abstract Recent advances in Knowledge Distillation (KD) aim to mitigate the high computational demands of Large Language Models (LLMs) by transferring knowledge from a large “teacher” to a smaller “student” model. However, students may inherit the teacher’s privacy when the teacher is trained on private data. In this work, we systematically characterize and investigate membership and memorization privacy risks inherent in six LLM KD techniques . Using instruction-tuning settings that span seven NLP tasks, together with three teacher model families (GPT-2, LLAMA-2, and OPT), and various size student models, we demonstrate that all existing LLM KD approaches carry membership and memorization privacy risks from the teacher to its students. However, the extent of privacy risks varies across different KD techniques . We systematically analyse how key LLM KD components (KD objective functions, student training data and NLP tasks) impact such privacy risks. We also demonstrate a significant disagreement between memorization and membership privacy risks of LLM KD techniques. Finally, we characterize per-block privacy risk and demonstrate that the privacy risk varies across different blocks by a large margin. Our code is available at https://github.com/ziqi-zhang/LLM_Distillation_Privacy . 1 Introduction Figure 1: Privacy risks of LLM KD. An adversary analyses membership and memorization of a teacher by only looking at its students. We quantify the performance of membership inference attacks in terms of AUC: the higher the value, the more the privacy risk. Distilling LLM knowledge reveals private membership information, the extent of this privacy risk varies across different knowledge distillation techniques. Knowldeg Distillation (KD) Hinton 2015 techniques have gained widespread adoption in practice 1 1 1 https://www.bbc.co.uk/news/articles/c9vm1m8wpr9o because of their performance and privacy benefits. KD reduces high computational costs and memory consumption of machine learning models Xu et al. 2024 . KD has been also recently adapted to protect the privacy leakage of LLMs Xiao et al. 2023 ; Tang et al. 2022 ; Mazzone et al. 2022 ; Shejwalkar and Houmansadr 2021 based on the assumption that distilling knowledge of a large model (teacher) through a public dataset to a small model (student) can protect the privacy of teacher’s training data. This privacy-preserving adaptation of KD has been gaining attraction as it gives the dual advantage of i) efficient students deployed on user devices and ii) better utility than provable protections promised by differentially private techniques. We examine the privacy risk in existing LLM KD techniques (see Figure 1 )– KD Hinton 2015 , SeqKD Kim and Rush 2016 , GKD Agarwal et al. 2024 , ImitKD Lin et al. 2020 , MiniLLM Gu et al. 2023 , and DistiLLM Ko et al. 2024 . We define and quantify the membership and memorization privacy leakage of the teacher’s private training data post-distillation . We comprehensively analyze whether membership information of the teacher’s training data can be inferred from its students: determine if a given data sample is in the teacher’s training set (member) or not (nonmember). We use seven Membership Inference Attacks (MIAs)– Min-K%++ Zhang et al. 2024 , Min-K% Shi et al. 2023 , Zlib Carlini et al. 2021 , LOSS Yeom et al. 2018 , and two reference-based attacks ( StableLM Duan et al. 2024 , Pretrain-Ref Fu et al. 2023 and MoPe ( Li et al. 2023 ) . As Figure 1 (right) shows, the adversary can still recover a large amount of membership information of the private data from the students. For instance, Pretrain-Ref recovers private membership information with a 0.83 Area Under the Curve (AUC) from student models obtained through ImitKD . The amount of recovered membership information varies across LLM KD techniques: from 0.64 ( GKD ) to 0.83 ( ImitKD ). We take the first step by measuring if and when teacher training data can be memorized by the student. In particular, we measure how much the student reproduces the teacher’s training data verbatim Carlini et al. 2023 . We find that students can memorize 11.35% of the same samples that the teacher memorizes. We also find that the student-memorized samples exhibit a different pattern with MIA. MIA is more effective in creative writing, general QA, brainstorming, and open QA than closed QA and classification, while the effectiveness of extraction attacks is vice versa. This finding demonstrates that memorization is not membership across KD tasks. We further study the source of the privacy leakage and its difference among LLM KD techniques by characterizing key components of KD. We create variants of students isolating the effect of each KD component, including the loss function and student-generated training dataset. We find that using reverse KL loss can mitigate MIA compared to KL. Utilizing student-generated data to stabilize the KD process will increase the MIA performance. We also reveal the privacy-utility-efficiency trade-off in KD : decreasing student size can improve privacy protection and efficiency, but harms utility. Finally, as an step towards designing LLM KD techniques with empirical privacy protections, we design a framework measuring a more fine-grained privacy leakage in LLMs. We first break LLMs into a sequence of transformer blocks. We then analyze each block’s privacy leakage by measuring the loss difference due to the model parameter perturbation similar to MoPe Li et al. 2023 . We demonstrate that privacy leakage varies across blocks within the same LLM . Take GPT2-Large as an example, the AUC of MIAs differs significantly from 0.50 (random guess; 34th block) to over 0.65 (5th block). We highlight the following contributions: • We define and comprehensively assess the membership privacy leakage of LLM KD using seven MIAs and six LLM KD techniques. • We define the memorization privacy leakage of LLM KD and present the first empirical study measuring whether training examples memorized by a teacher model remain memorized by the student model after KD. • We present a per-block privacy analysis framework and reveal that privacy leakage varies across blocks. 2 Problem formulation As shown in Fig. 1 , we consider training a teacher model \mathcal{M} on private data \mathcal{D}_{\text{Private}} . \mathcal{M} consists of a set of transformer blocks Vaswani et al. 2017 with a huge number of parameters \theta=\{\theta^{l}\}_{l=1}^{L} . Each \theta^{l} represents parameters of l -th block. Privacy Risk and Computation Costs. An institution needs to solve two issues when it wants to deploy a LLM to users’ devices. The first issue is the privacy risk introduced for \mathcal{D}_{\text{Private}} as \mathcal{M} might memorize and unintentionally leak information about their training data Carlini et al. 2023 ; Biderman et al. 2024 . A malicious user can utilize the memorization phenomenon to recover the private information in \mathcal{D}_{\text{Private}} . The second issue is the high computation cost. LLMs require many parameters to perform complex operations and cost computation resources. For example, the state-of-the-art LLMs such as GPT-3 and GPT-4 contain over 100B parameters Liu et al. 2024 . 2.1 Knowledge Distillation for LLMs Knowledge Distillation (KD) techniques Xiao et al. 2023 ; Xu et al. 2024 ; Hinton 2015 ; Kim and Rush 2016 ; Agarwal et al. 2024 ; Lin et al. 2020 ; Gu et al. 2023 ; Ko et al. 2024 have been proposed to address above issues–privacy

Read the complete text