Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert · 2026

Read Unintended Memorization of Sensitive Information in Fine-Tuned Language Models by Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17480.txt.

Introduction

Marton Szep and five co-authors study a privacy leak in fine-tuned language models. Fine-tuning on sensitive data can make models memorize it. Names and other personal facts can then leak out. The team looks at a less studied path. They test facts that appear only in model inputs, not in the target outputs.

Input-only facts are easy to overlook in safety checks. If models store them, chat logs and forms are at risk. The study maps how bad that exposure can be. Its lessons guide data handling for health and legal tools. Teams learn what to strip before training starts. Privacy law sets a high bar here, and leaks can breach it.

Catalog record

CREATORMarton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17480

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unintended Memorization of Sensitive Information in Fine-Tuned Language Models Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert Abstract Fine-tuning Large Language Models (LLMs) on sensitive datasets carries a substantial risk of unintended memorization and leakage of Personally Identifiable Information (PII), which can violate privacy regulations and compromise individual safety. In this work, we systematically investigate a critical and underexplored vulnerability: the exposure of PII that appears only in model inputs, not in training targets. Using both synthetic and real-world datasets, we design controlled extraction probes to quantify unintended PII memorization and study how factors such as language, PII frequency, task type, and model size influence memorization behavior. We further benchmark four privacy-preserving approaches including differential privacy, machine unlearning, regularization, and preference alignment, evaluating their trade-offs between privacy and task performance. Our results show that post-training methods generally provide more consistent privacy–utility trade-offs, while differential privacy achieves strong reduction in leakage in specific settings, although it can introduce training instability. These findings highlight the persistent challenge of memorization in fine-tuned LLMs and emphasize the need for robust, scalable privacy-preserving techniques. 1 Introduction and Related Work Large Language Models (LLMs) achieve state-of-the-art performance across numerous natural language processing tasks, but their vast capacity and data-hungry training regimes raise serious privacy concerns. Most notably, LLMs can memorize training samples even if seen only once ( 2 ) . While some memorization supports generalization in long-tailed data distributions ( 6 ) , verbatim token-level memorization of Personally Identifiable Information (PII) poses significant privacy risks. Figure 1: Overview of our experiment setup depicting the unintended PII memorization scenario, our attack, and fine-tuning approaches. Prior studies have analyzed memorization dynamics in LLMs during both pre-training (PT) and fine-tuning (FT) ( 14 ; 2 ; 10 ) . For example, 1 examined how model size, data duplication, and prompt length influence memorization during PT, and 26 studied task-specific memorization during FT. These works primarily focus on task-relevant data. In practice, however, FT often (inadvertently) involves inputs containing sensitive PII that are unrelated to the task output , such as names or medical records. Despite growing attention to LLM privacy, the specific risk of unintended PII memorization , where PII appears only in inputs and is irrelevant to the downstream task, remains largely unexplored. Recent studies have highlighted PII leakage in realistic deployment scenarios, including black-box probing and adversarial API querying ( 16 ; 15 ; 13 ) . Yet, no prior work systematically isolates input-only PII memorization or compares mitigation strategies across methods. In this work, we present a comprehensive study addressing this gap. Specifically, we: • Define and formalize the problem of input-only PII memorization, distinguishing it from general memorization or task-relevant PII usage; • Quantify memorization using synthetic and real-world datasets in realistic deployment scenarios; • Analyze key factors influencing memorization severity, including language, PII repetition, model capacity, prefix length, and downstream task type; • Benchmark four common mitigation strategies, assessing their privacy-utility tradeoffs: differential privacy, regularization, machine unlearning, and preference alignment. To the best of our knowledge, this is the first comprehensive study focused on unintended PII memorization in LLM fine-tuning, highlighting challenges and considerations for privacy-preserving model deployment. 2 Methodology 2.1 Unintended, Input-only PII Memorization We define unintended PII memorization as the phenomenon in which a language model fine-tuned on sensitive text data, such as electronic health records (EHRs), internalizes PII that is not part of its intended output (i.e., unrelated to the downstream task). This is distinct from memorization during pre-training, where large corpora might contain public or semi-public PII, and from targeted FT tasks, where PII is intentionally part of the model’s output space. Our work focuses on downstream tasks (classification, information extraction, medical follow-up planning) in which PII appears only in the inputs, not in the training targets. We adopt a realistic black-box threat model where adversaries access the model only via input-output queries (e.g., API calls). We assume a worst-case scenario where attackers have partial access to the FT dataset (e.g. anonymized EHRs) and can craft adversarial prompts accordingly ( 1 ; 16 ) . True-Prefix Attack (TPA) is a method to probe memorization in autoregressive LLMs ( 2 ) . Given a true prefix c from the FT data immediately preceding a PII span s of N tokens, we say s is extractable if s\leftarrow\underset{s^{\prime}:\,|s^{\prime}|=N}{\arg\max}\,f_{\theta}(s^{\prime}\mid c). (1) where f_{\theta}(s^{\prime}\mid c) is the model’s conditional probability distribution. With labeled PII spans, this attack is straightforward to construct and evaluate, providing an effective measure of model memorization. Throughout the paper, we use TPA as the main PII extraction method and also define an enhanced TPA variant, which adds the first character of the PII to the prefix. We also compare the true-prefix attack to other adversarial attacks employing more sophisticated prompting techniques in § A.3 , and find that the extraction rate achieved by TPA serves as an approximate lower bound. 2.2 Mitigating memorization We evaluate four prevalent training strategies aimed at reducing PII memorization during or after FT. Differential Privacy (DP) is a widely used technique for protecting individual data privacy with mathematical guarantees ( 11 ; 4 ) . It introduces noise into the gradient updates and limits individual sample influence, thus bounding sample-level memorization risk. DP has been extensively applied to both LLM pre - training and fine - tuning, providing verifiable guarantees, but at the cost of utility degradation and increased training complexity ( 9 ; 12 ; 25 ) . Machine Unlearning: UnDial ( 3 ) is a targeted unlearning method based on self-distillation. It constructs an adjusted target distribution by suppressing the logits of tokens associated with the information to be forgotten. By relying on self-distillation rather than directly maximizing loss on the forget set, UnDial mitigates training instability and reduces unintended degradation of general model performance, issues observed in earlier unlearning methods such as Gradient Ascent and Negative Preference Optimization ( 5 ; 19 ) . Regularization Inspired by UnDial ( 3 ) , we propose a regularization-based variant that integrates self-distillation into the FT loop. Specifically, we alternate between cross-entropy loss and a regularization loss focused on PII tokens. This focused UnDial loss is applied only on selected sensitive spans to discourage memorization. Direct Preference Optimization (DPO) emerges as a computationally and data-efficient alternative to RLHF for aligning models’ outputs with human preferences, such as privacy or helpfulness ( 18 ; 20 ) . We adapt DPO to discourage PII leakage by treating training examples that contain PII as rejected and their corresponding masked versions as preferred . 3 Experiments In our experiments, we primarily focus on extracting memorized personal names, as they constitute the most heterogeneous and unstructured category of PII and defy rule-based detection using regular expressions. Our extended analysis across PII types in § A.4 reveals that while SFT universally exacerbates leakage across all PII types in synthetic data, it yields mixed results in real-world domains. Specifically, we find that fine-tuning can reduce the extraction of generic structured priors (e.g., dates, postal codes) compared to the base model, in contrast to the persistent memorization risks observed for unstructured identifiers. 3.1 Datasets We use three datasets varying significantly in nature, task complexity,

Read the complete text