Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models
2026
Read Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29497.txt.
Introduction
Loiseau and co-authors distill privacy judgment from a large language model into small encoders. Rating text for privacy risk matters for privacy-preserving language tools. Big models judge privacy well but cost too much at scale. They also cannot easily process sensitive data. The team trains light classifiers on privacy-labeled texts from ten domains. The teacher is Mistral Large 3, a very large model. The students have as few as 150 million parameters. They keep strong agreement with human labels at far lower cost.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31676 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29497
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models Abstract Accurate privacy evaluation of textual data remains a critical challenge in privacy-preserving natural language processing. Recent work has shown that large language models (LLMs) can serve as reliable privacy evaluators, achieving strong agreement with human judgments; however, their computational cost and impracticality for processing sensitive data at scale limit real-world deployment. We address this gap by distilling the privacy assessment capabilities of Mistral Large 3 (675B) into lightweight encoder models with as few as 150M parameters. Leveraging a large-scale dataset of privacy-annotated texts spanning 10 diverse domains, we train efficient classifiers that preserve strong agreement with human annotations while dramatically reducing computational requirements. We validate our approach on human-annotated test data and demonstrate its practical utility as an evaluation metric for de-identification systems. Keywords: privacy evaluation, knowledge distillation, de-identification Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models Gabriel Loiseau 1,2 , Damien Sileo 2 , Damien Riquet 1 , Maxime Meyer 1 , Marc Tommasi 2 1 Hornetsecurity, Hem, France 2 Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France [email protected] Abstract content 1. Introduction Quantifying privacy in textual data remains an open challenge due to the absence of a unified definition and the inherently contextual nature of privacy ( Bambauer et al. 2022 ; Tesfay et al. 2016 ) . Formal frameworks such as differential privacy ( Dwork 2006 ) provide rigorous guarantees, and proxy-based evaluation through attack success rates or information-theoretic measures are well-established in practice Ren et al. 2025 . However, these approaches capture specific, well-defined threat models rather than the broader, human-perceived notion of what constitutes sensitive content. Large language models (LLMs), with their capacity for nuanced language understanding, have emerged as promising candidates for human-aligned evaluation, demonstrating strong agreement with human judgments across a variety of NLP tasks ( Zheng et al. 2023 ; Li et al. 2024 ) . Rating Name Description Dataset (%) 1 Harmless Completely free of any private or sensitive information, including direct or indirect identifiers. 46.01 2 Mostly not private May contain some indirect identifiers, but is largely free of sensitive or personal information. 16.58 3 Somewhat private Contains some direct or indirect identifiers and can be considered moderately personal. 16.81 4 Very private Contains several direct or indirect identifiers and clearly includes personal information. 14.21 5 Extremely private Contains highly sensitive personal information or direct identifiers. 6.38 Table 1: Privacy rating annotation scheme and resulting dataset distribution Recent work by Meisenbacher et al. 2025 represents a significant step toward closing this gap in privacy evaluation by applying the LLM-as-a-Judge paradigm to this domain. Across 10 datasets and 677 human annotators, they show that LLMs can approximate a “global human privacy perspective” with strong agreement to aggregated human ratings, even exceeding inter-human agreement. These findings suggest that LLMs can serve as practical, human-aligned privacy evaluators. Yet, deploying frontier LLMs for privacy assessment poses two central challenges. First, their computational and financial costs limit large-scale use. Second, evaluating sensitive text through third-party APIs introduces additional privacy concerns, as the very data being assessed may not be shareable. This creates a paradox: using powerful external LLMs to evaluate privacy may itself compromise privacy constraints. In this work, we address this deployment gap through knowledge distillation . Using Mistral Large 3 ( Mistral AI 2025 ) as a teacher model, we annotate 200,000 user-written texts with privacy sensitivity scores following the structured Likert-scale methodology of Meisenbacher et al. 2025 . We then distill these judgments into lightweight encoder-based classifiers, enabling fast, local, and privacy-preserving inference. Our central research question is whether the privacy reasoning capabilities of LLMs can be transferred to smaller models without sacrificing alignment with human judgments. We validate the distilled models on human-annotated test data and show that they can match the agreement of their teacher model with aggregated human ratings. Beyond benchmark validation, we demonstrate that distilled privacy evaluators can serve as scalable automatic metrics for quantifying privacy reduction in text de-identification systems 1 1 1 Models, code, and data are available at https://github.com/gabrielloiseau/privacy-distillation . Our contributions are threefold: 1. We curate a large corpus of 200,000 texts, automatically annotated for privacy sensitivity using a state-of-the-art open LLM. 2. We distill these LLM-generated privacy judgments into lightweight encoder models that achieve strong agreement with human annotations ( \alpha=0.737 ), surpassing the teacher model’s own human alignment ( \alpha=0.716 ), while enabling efficient and fully local inference. 3. We demonstrate that distilled privacy evaluators function as scalable automatic metrics for assessing privacy reduction in text de-identification systems, and outline how compact privacy models open new research directions for privacy-aware NLP evaluation and system design. 2. Related Work Privacy Evaluation in NLP. In privacy-preserving NLP, evaluation commonly relies on proxy metrics such as re-identification success rates, simulated attacks, plausible deniability, or semantic similarity measures ( Shahriar et al. 2025 ) . While these metrics capture specific threat models, they do not directly reflect how humans perceive the sensitivity of a text. Complementary research on automated privacy policy analysis ( Wilson et al. 2016 ) and anonymization benchmarks ( Lison et al. 2021 ; Pilán et al. 2022 ; Loiseau et al. 2025 ) provides structured evaluation frameworks, primarily focusing on entity-level redaction quality. However, these approaches do not measure text-level privacy sensitivity or its alignment with human judgment across domains. Recent work has proposed LLM-as-a-Judge as a scalable alternative to human evaluation for many NLP tasks ( Li et al. 2024 ; Chiang and Lee 2023 ; Bavaresco et al. 2025 ; Li et al. 2025 ) , with potential for modeling perceived privacy risk ( Meisenbacher et al. 2025 ) . LLM Distillation. Knowledge distillation ( Hinton et al. 2015 ) transfers capabilities from large teacher models to smaller student models. In NLP, it has produced efficient transformer variants such as DistilBERT ( Sanh et al. 2019 ) and compressed generative LLMs into lightweight classifiers. Distillation can also rely solely on predicted labels, enabling black-box knowledge transfer when logits are unavailable ( Chen et al. 2024 ) . 3. Methodology 3.1. Privacy Annotation Framework We adopt the five-point Likert privacy sensitivity scale introduced by Meisenbacher et al. 2025 and detailed in Table 1 , ranging from 1 ( Harmless ) to 5 ( Extremely private ). The scale operationalizes text-level privacy sensitivity by considering both direct identifiers (e.g., names, contact details) and broader contextual signals, including topical sensitivity (e.g., health conditions, legal situations), self-disclosure of personal experiences, and indirect identifiers that could enable re-identification in context. Sensitivity under this scale is therefore not limited to the presence of named entities or demographic attributes, but also encompasses the overall nature and intimacy of the disclosed content. The scale was previously validated through a large-scale human annotation study, and the resulting survey data were publicly released, providing a human-aligned target for supervision. 3.2. LLM Annotation and Distillation Pipeline Data. We construct a corpus from the 10 publicly available datasets of user-written text from the original study spanning diverse domains: Blog Authorship Corpus (BAC), Enron Emails (EE), Medical Questions (MQ), Reddit Confessions (RC), Reddit Legal Advice (RLA), Mental Health Blog (MHB), Reddit Mental Health Posts (RMHP), Trustpilot Reviews