Can We Infer Confidential Properties of Training Data from LLMs?
Pengrun HuangUC San Diego [email protected] Chhavi YadavUC San Diego [email protected], Ruihan WuUC San Diego [email protected], Kamalika ChaudhuriUC San Diego [email protected] · 2025
Read Can We Infer Confidential Properties of Training Data from LLMs? by Pengrun HuangUC San Diego [email protected] Chhavi YadavUC San Diego [email protected], Ruihan WuUC San Diego [email protected], Kamalika ChaudhuriUC San Diego [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10364.txt.
Introduction
Huang, Yadav, Chaudhuri, and Wu asked in twenty twenty-five whether word models leak group facts of their fine-tune data, like who the patients were. Firms tune models on health, money, and law sets with hush facts baked in. Past leak tests hit image models, not word ones.
They built a bench task for such leaks on a doctor chat set. Two attacks were tried: a prompt push and a twin-model read of word rates. Both pulled group facts out of tuned models in their runs. The hole is real and was not known for word models, they warn.
Catalog record
| CREATOR | Pengrun HuangUC San Diego [email protected] Chhavi YadavUC San Diego [email protected], Ruihan WuUC San Diego [email protected], Kamalika ChaudhuriUC San Diego [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-10364
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can We Infer Confidential Properties of Training Data from LLMs? Pengrun HuangUC San Diego [email protected] Chhavi YadavUC San Diego [email protected], Ruihan WuUC San Diego [email protected], Kamalika ChaudhuriUC San Diego [email protected] Abstract Large language models (LLMs) are increasingly fine-tuned on domain-specific datasets to support applications in fields such as healthcare, finance, and law. These fine-tuning datasets often have sensitive and confidential dataset-level properties — such as patient demographics or disease prevalence—that are not intended to be revealed. While prior work has studied property inference attacks on discriminative models (e.g., image classification models) and generative models (e.g., GANs for image data), it remains unclear if such attacks transfer to LLMs. In this work, we introduce PropInfer, a benchmark task for evaluating property inference in LLMs under two fine-tuning paradigms: question-answering and chat-completion. Built on the ChatDoctor dataset, our benchmark includes a range of property types and task configurations. We further propose two tailored attacks: a prompt-based generation attack and a shadow-model attack leveraging word frequency signals. Empirical evaluations across multiple pretrained LLMs show the success of our attacks, revealing a previously unrecognized vulnerability in LLMs. 1 1 footnotetext: † Equal advising. 1 Introduction Large language models (LLMs) are increasingly deployed in real-world applications across domains such as healthcare he2025surveylargelanguagemodels , finance li2024largelanguagemodelsfinance , and law lai2023largelanguagemodelslaw . To adapt to domain-specific tasks, such as customer service or tele-medicine, these models are typically fine-tuned on proprietary datasets that are relevant to the tasks at hand before deployment. These domain-specific fine-tuning datasets however often contain dataset-level confidential information . For example, a customer-service dataset sourced from a business may contain information about their typical customer-profile; a doctor-patient chat dataset sourced from a hospital may contain patient demographics or the fraction of patients with a sensitive disease such as HIV. Many businesses and medical practices would consider this kind of information non-public for business or other reasons. Thus, unintentional leakage of this information through a deployed model could lead to a breach of confidentiality. Unlike individual-level privacy breaches that is typically addressed by rigorous definitions such as differential privacy ( dwork2006calibrating ; dwork2014algorithmic ) , the risk here is the leakage of dataset-level properties. Prior work has investigated this form of leakage, commonly referred to as property inference ( ateniese2013hackingsmartmachinessmarter ) . Most of the literature here has focused on two settings. The first involves discriminative models trained on tabular or image data ( ateniese2013hackingsmartmachinessmarter ; ganju2018property ; chase2021propertyinferencepoisoning ; suri2024dissectingdistributioninference ; zhang2021leakagedatasetpropertiesmultiparty ; 10136150 ) , where the goal is to infer attributes such as the gender distribution in a hospital dataset. The second focuses on generative models ( zhou2021propertyinferenceattacksgans ; wang2024property ) , such as GANs for face synthesis, where attackers may attempt to recover aggregate properties such as the racial composition of the training data. In both cases, property inference has been shown to be feasible, and specialized attacks have been proposed to exploit these vulnerabilities. However, property inference in large language models (LLMs) introduces two distinct challenges. First, unlike inferring a single attribute from models trained on tabular data, the sensitive properties are more complex and may be indirectly embedded within the text. For example, gender might be implied through broader linguistic cues, such as the mention of a “my gynecologist”. LLMs may memorize such properties implicitly, making them more challenging to infer reliably in property inference studies. The second challenge is that, unlike the models typically studied in prior work, LLMs do not fit cleanly into purely discriminative or generative categories; this raises questions about what kind of property inference attacks apply and succeed for these problems. In this work, we investigate both questions by introducing a new benchmark task -- PropInfer 1 1 1 https://huggingface.co/datasets/Pengrun/PropInfer_dataset – for property inference in LLMs. Our task is based on the Chat-Doctor dataset ( li2023chatdoctormedicalchatmodel ) – a domain-specific medical dataset containing a collection of question-answer pairs between patients and doctors. There are two standard ways to fine-tune an LLM with this dataset that correspond to two use-cases: question-answering and chat-completion. According to the use case, our benchmark task has two modes where the models are fine-tuned differently – Q&A Mode and Chat-Completion Mode. To comprehensively study property inference across the two modes of models, we select a range of properties that are explicitly or implicitly reflected in both questions and answers. We propose two property inference attacks tailored to LLMs. The first is a black-box generation-based attack, inspired by prior work ( zhou2021propertyinferenceattacksgans ) ; the intuition is that the distribution of the generated samples reflect the distribution of the fine-tuning data. Given designed prompts that reflects characteristics of the target dataset, the adversary generates multiple samples from the target LLM and labels each based on the presence of the target property. The property ratio is then estimated by aggregating the labels. The second is a shadow-model attack with word-frequency. With access to an auxiliary dataset, the adversary first trains a set of shadow models with varying property ratios and extracts word frequency from the shadow models based on some selected keyword list. Then the adversary trains a meta-attack model that maps these frequencies to the corresponding property ratios. This enables the inference on the target model by computing its output word frequencies. We empirically evaluate our two attacks alongside baseline methods using our PropInfer-benchmark. Our results show that the shadow-model attack with word frequency is particularly effective when the target model is fine-tuned in the Q&A Mode and the target property is more explicitly revealed in the question content than the answer. In contrast, when the model is fine-tuned in Chat-Completion Mode or when the target attribute are embedded in both the question and the answer, the black-box generation-based attack proves to be simple yet highly effective. Our experimental results reveal a previously underexplored vulnerability in large language models: property inference, which enables adversaries to extract dataset-level attributes from fine-tuned models. This finding exposes a tangible threat to data confidentiality in real-world deployments. It also underscores the need for robust defense mechanisms to mitigate such attacks – an area where our benchmark provides a standardized and extensible framework for future research and evaluation. 2 Related Work Property inference: Property Inference Attack (PIA) was first described by ateniese2013hackingsmartmachinessmarter , as follows: given two candidate training data distributions \mathcal{D}_{1},\mathcal{D}_{2} and a target model, the adversary tries to guess which training distribution (out of \mathcal{D}_{1},\mathcal{D}_{2} ) is the target model trained on. Typically, the two candidate distributions only differ in the marginal distribution of a binary variable, such as gender ratio. A major portion of past work on property inference focuses on discriminative models ateniese2013hackingsmartmachinessmarter ; ganju2018property ; chase2021propertyinferencepoisoning ; suri2024dissectingdistributioninference ; zhang2021leakagedatasetpropertiesmultiparty ; 10136150 ; here the attacks mainly rely on training meta-classifiers on some representations to predict target ratio. For example, in the white-box setting, ateniese2013hackingsmartmachinessmarter ; ganju2018property use model weights as the input of the meta-classifier to predict the correct distribution. In the grey-box setting, where the adversary have access to the training process and some auxiliary data, suri2022formalizingestimatingdistributioninference ; suri2024dissectingdistributioninference use model outputs such as loss or probability vector as inputs to the meta-classifier. Moving on to generative models, zhou2021propertyinferenceattacksgans study property inference attack for GANs. The target GANs are