RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis

Jianwei Wang, Chengming Shi, Junyao Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng · 2025

Read RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis by Jianwei Wang, Chengming Shi, Junyao Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18517.txt.

Description

Hospitals and banks hold exactly the data that would fine-tune a language model well — and exactly the data they cannot upload. Synthetic stand-ins generated with differential privacy are the usual workaround. But much of the synthetic output is flawed. Crude filters based on text overlap or embedding similarity keep too much of the noise.

Jianwei Wang and co-authors present RewardDS in 2025. A small reward model, itself fine-tuned privately, scores the synthetic data. One module filters by those scores. A second refines borderline examples in a self-optimizing loop, using the reward signal as the guide. Across medical, financial, and code tasks, the cleaned data trained better models than rival filtering schemes. The authors offer the framework as a practical way to fine-tune on sensitive domains without the sensitive data ever leaving home.

Catalog record

CREATORJianwei Wang, Chengming Shi, Junyao Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18517

Browse the full library · About the Tower of Babel