Differentially Private Preference Data Synthesis for Large Language Model Alignment
Fengyu Gao, Jing Yang · 2026
Read Differentially Private Preference Data Synthesis for Large Language Model Alignment by Fengyu Gao, Jing Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30808.txt.
Introduction
Gao and Yang propose DPPrefSyn, generating private synthetic preference data for large language model alignment. Post-training on real preference data raises privacy concerns, as datasets contain sensitive prompts and judgments. DPPrefSyn is grounded in the Bradley-Terry model and pairwise preference geometry. It first learns a preference model from private data, then uses it with public prompts to synthesize data. It exploits shared linear structure across reward models to capture varied preferences, and uses private component analysis. Tests show DPPrefSyn achieves competitive alignment performance under strong privacy guarantees.
Catalog record
| CREATOR | Fengyu Gao, Jing Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94531 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30808
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Differentially Private Preference Data Synthesis for Large Language Model Alignment Fengyu Gao, Jing Yang Abstract Preference alignment is a crucial post-training step for large language models (LLMs) to ensure their outputs align with human values. However, post-training on real human preference data raises privacy concerns, as these datasets often contain sensitive user prompts and human judgments. To address this, we propose DPPrefSyn , a novel algorithm for generating differentially private (DP) synthetic preference data to enable privacy-preserving preference alignment. DPPrefSyn is a principled framework grounded in the Bradley–Terry preference model and the intrinsic geometric structure of pairwise human preference data. It first learns an underlying preference model from private data with formal differential privacy guarantees, and then leverages the learned model together with public prompts to synthesize high-quality preference data. It exploits the shared linear structure of per-cluster reward models to effectively capture heterogeneous human preferences in private datasets, and leverages DP Principal Component Analysis (DP-PCA) to improve learning accuracy. Extensive experimental results demonstrate that DPPrefSyn achieves competitive alignment performance under strong DP guarantees. These findings highlight the potential of synthetic preference data as a practical alternative for privacy-preserving preference alignment across a broad range of applications. To the best of our knowledge, this is the first work to generate DP synthetic preference data for LLM alignment. Our code is available at https://github.com/gfengyu/Differentially-Private-Preference-Data-Synthesis . 1 Introduction Preference alignment techniques, such as RLHF ( 59 ) and DPO ( 48 ) , are widely used to align LLMs with human expectations. These algorithms rely on pairwise preference data, where human annotators compare two responses and select the one that better addresses the given prompt. Such preference data is then used to fine-tune LLMs by encouraging the model to rank preferred responses over less preferred ones ( 82 ; 46 ; 6 ) . The effectiveness and reliability of preference alignment have motivated its adoption in applications such as chat assistants ( 2 ) , mathematical reasoning tools ( 53 ) , and code generators ( 54 ) . However, aligning LLMs with human preference data raises substantial privacy concerns, since these datasets frequently include real user prompts and human feedback. These prompts may disclose personal information related to health, identity, or other sensitive topics, and human feedback may reveal private beliefs, preferences, or behavioral patterns ( 37 ; 75 ) . Existing works ( 14 ; 75 ; 68 ) have explored approaches to mitigate these risks using the rigorous privacy safeguards provided by differential privacy (DP) ( 18 ) and shown encouraging results. However, many of these approaches protect only part of the data, for example, by privatizing either the user prompts ( 75 ) or the preference labels ( 14 ; 78 ) , but not both. Such partial protection leaves potential privacy gaps, as adversaries could still infer sensitive attributes from unprotected prompts or preference annotations. Besides, such works usually focus on specific post-training algorithms, such as RLHF ( 68 ) , and are not compatible with newer methods like DPO. Moreover, these methods are typically constrained by limited private preference datasets, and often fall short in achieving high-quality preference alignment, due to the high cost and scalability challenges associated with collecting human preference annotations. These gaps lead to the central question of this work: Is it possible to achieve superior preference alignment while ensuring thorough privacy protection and maintaining compatibility with emerging alignment methodologies? In this work, we provide an affirmative answer to this question. Our main contributions are summarized as follows: • We introduce a new algorithm DPPrefSyn ( Figure 1 ) to generate DP synthetic preference data for LLM alignment. DPPrefSyn is a principled framework grounded in the Bradley–Terry preference model and the intrinsic geometric structure of pairwise human preference data. It features the following design elements: 1) Capturing diverse human preferences via shared linear structure of rewards and clustering. The Bradley–Terry preference model, together with a linear reward structure, enables us to group preference samples based on embedding differences between concatenated prompt–preferred and prompt–dispreferred pairs, and to approximate preferences within each cluster using a shared reward function. 2) Improving learning efficiency through DP-PCA. To address sample inefficiency and clustering instability in high-dimensional spaces, we apply DP-PCA to project the embeddings into a lower-dimensional space while preserving key preference signals under DP. 3) Saving privacy budgets by using public prompts. We use public prompts to generate candidate responses and apply private reward models to select preference pairs. This allows us to focus the privacy budget entirely on modeling private preferences, improving privacy efficiency. • We conduct a rigorous privacy analysis of DPPrefSyn and confirm it follows (\varepsilon,\delta) -DP. In our analysis, we first allocate a privacy budget \varepsilon_{0} to DP-PCA ( 3 ) for dimensionality reduction, and \varepsilon_{1} to DP-KMeans ( 60 ) for clustering. The remaining budget is used to train per-cluster reward models with DP-SGD. We apply the Privacy Random Variable (PRV) accountant ( 23 ) to tightly compose the DP-SGD privacy cost, ensuring that DPPrefSyn satisfies (\varepsilon,\delta) -DP. Thanks to the post-processing property of DP, the resulting DP synthetic dataset can be reused across multiple preference alignment methods and different LLMs. • We empirically evaluate DPPrefSyn on standard benchmarks, including question answering tasks from OpenAssistant ( 33 ) and Anthropic-HH ( 6 ) , as well as the TL;DR summarization task ( 59 ) . Our experiments suggest that synthetic preference data generated by DPPrefSyn offers strong privacy guarantees while achieving competitive utility across diverse tasks. For example, with \varepsilon=2 , fine-tuning the Pythia-2.8B model using DPO on our synthetic data achieves a GPT-4o win rate of 56.48\% on Anthropic-HH, significantly outperforming fine-tuning on real data ( 37.02\% ). These results show that our DP synthetic preference data can enable effective preference alignment of LLMs with strong privacy guarantees. Moreover, DPPrefSyn can be applied to various preference optimization methods (e.g., DPO and RLHF) and remains effective under strong privacy constraints, overcoming the limitations of other privacy-preserving post-training approaches that are tailored to specific methods. Figure 1 : Overview of DPPrefSyn. DPPrefSyn generates DP synthetic preference data through 3 steps: 1) representing preference samples as embedding differences and clustering them via DP-PCA and DP-KMeans, 2) training DP reward models on each cluster using DP-SGD, and 3) generating synthetic preference samples from public prompts guided by the DP-protected distribution of reward functions. 2 Related Work Differentially Private Alignment of LLMs. 68 introduce a DP framework to align LLMs with reinforcement learning by adapting the PPO algorithm to the DP setting. However, their approach is limited to PPO , while preference alignment continues to evolve with new methods. 14 study the problem of reward estimation from preference-based feedback, using the notion of label-DP ( 22 ) to protect the privacy of human annotators. 78 propose AUP-RLHF, a user-level label-DP framework for RLHF. While both methods are effective at safeguarding preference labels, they do not address the sensitivity of prompts or responses themselves . Besides, such methods are usually constrained by the limited private user preference data , and may not achieve high-quality preference alignment, as human preference annotations are costly to obtain. 75 focus on protecting sensitive user instructions by generating DP synthetic instructions to