RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis
Jianwei Wang, Chengming Shi, Junyao Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng · 2025
Read RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis by Jianwei Wang, Chengming Shi, Junyao Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18517.txt.
Description
Hospitals and banks hold exactly the data that would fine-tune a language model well — and exactly the data they cannot upload. Synthetic stand-ins generated with differential privacy are the usual workaround. But much of the synthetic output is flawed. Crude filters based on text overlap or embedding similarity keep too much of the noise.
Jianwei Wang and co-authors present RewardDS in 2025. A small reward model, itself fine-tuned privately, scores the synthetic data. One module filters by those scores. A second refines borderline examples in a self-optimizing loop, using the reward signal as the guide. Across medical, financial, and code tasks, the cleaned data trained better models than rival filtering schemes. The authors offer the framework as a practical way to fine-tune on sensitive domains without the sensitive data ever leaving home.
Catalog record
| CREATOR | Jianwei Wang, Chengming Shi, Junyao Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-18517