Privacy-Preserving Model Transcription with Differentially Private Synthetic Distillation

Bochao Liu, Shiming Ge, Pengju Wang, Shikun Li, Tongliang Liu · 2026

Read Privacy-Preserving Model Transcription with Differentially Private Synthetic Distillation by Bochao Liu, Shiming Ge, Pengju Wang, Shikun Li, Tongliang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19090.txt.

Introduction

Bochao Liu and four co-authors copy a trained model into a new one without leaking its private data. Models trained on private sets are shipped widely, but attackers can mine a shipped model for facts about its training data or labels. The team offers a conversion that carries a formal privacy guarantee.

Their method, called model transcription, is data free: it needs no access to the raw private set. It uses synthetic data, made private by design, to teach the copy. A safe way to reissue models would ease deployment in health, finance, and other fields where data cannot travel.

Catalog record

CREATORBochao Liu, Shiming Ge, Pengju Wang, Shikun Li, Tongliang Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19090

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Privacy-Preserving Model Transcription with Differentially Private Synthetic Distillation Bochao Liu, Shiming Ge, Pengju Wang, Shikun Li, Tongliang Liu Abstract While many deep learning models trained on private datasets have been deployed in various practical tasks, they may pose a privacy leakage risk as attackers could recover informative data or label knowledge from models. In this work, we present privacy-preserving model transcription , a data-free model-to-model conversion solution to facilitate model deployment with a privacy guarantee. To this end, we propose a cooperative-competitive learning approach termed differentially private synthetic distillation that learns to convert a pretrained model (teacher) into its privacy-preserving counterpart (student) via a trainable generator without access to private data. The learning collaborates with three players in a unified framework and performs alternate optimization: i) the generator is learned to generate synthetic data, ii) the teacher and student accept the synthetic data and compute differential private labels by flexible data or label noisy perturbation, and iii) the student is updated with noisy labels and the generator is updated by taking the student as a discriminator for adversarial training. We theoretically prove that our approach can guarantee differential privacy and convergence. The transcribed student has good performance and privacy protection, while the resulting generator can generate private synthetic data for downstream tasks. Extensive experiments clearly demonstrate that our approach outperforms 26 state-of-the-arts. I Introduction Deep learning models [ 40 , 63 , 33 , 18 ] are widely used to learn rich knowledge from substantial real-world datasets, subsequently deployed in various critical tasks. For instance, a three-layer fully-connected net was trained on the extensive binary classification dataset SocialEvent for abnormal detection via trustable co-label learning and gave a validate AUC of 75.21% [ 43 ] , while a ResNet64 model was trained on massive personal identity photos for ID versus Spot face recognition via large-scale bisample learning and achieved a verification rate of 96.43% at low false acceptance rate [ 89 ] . Beyond their success, there is growing concern about their potential to compromise privacy from training data. This could occur through model inversion attacks with limited access [ 27 , 78 ] or privacy violation via inference with large language models [ 58 , 67 ] , especially as these models become accessible [ 64 ] , as shown in Fig. 1 . Therefore, high-accuracy models with a privacy guarantee are critical for real-world deployment. To this end, exploring a feasible solution that can address a key challenge in model re-deployment is necessary: how to effectively convert a well-trained model into a privacy-preserving one without significant loss of inference accuracy? Fig. 1: A model T directly trained on private data may leak privacy in practical deployment, which inspires us to propose model transcription that can convert T into a privacy-preserving model S without access to private data. An intuitive idea for model re-deployment is model compression [ 16 ] whose main objective is to reduce model parameters or storage so that the model can be deployed into resource-limited devices. Existing approaches primarily rely on network quantization, network pruning, and knowledge distillation. Network quantization and pruning reduce the model parameters by low-precision approximation [ 68 , 72 ] and redundancy removal [ 49 , 81 ] , respectively, which mitigates model sensitivity to small data change. In contrast, knowledge distillation [ 31 , 14 , 24 ] involves using a pretrained model as teacher to train a more compact student model by behavior mimicking. Such solution allows the distilled student to avoid direct access to private data by training on public [ 70 ] or generated data [ 53 , 11 ] . However, evaluating the privacy protection capability of these model compression techniques is challenging. A common strategy for privacy evaluation is differential privacy [ 20 ] , which aims to learn models on private data while protecting data privacy [ 1 ] and has been modified to achieve label privacy [ 8 , 30 , 22 ] . Nonetheless, when private data are unavailable, applying differential privacy directly to learn private models is impractical. To overcome this, some works [ 59 , 39 , 28 ] have integrated knowledge distillation with differential privacy, using an ensemble of pretrained models as teachers to learn private models on public data. A recent model conversion approach [ 48 ] generated synthetic data and then exploited them to distill pretrained models into privacy-preserving ones without access to private data. These approaches demonstrate the potential of combining knowledge distillation and differential privacy for private model training without original datasets but struggle to distill the rich knowledge from teachers to students with a privacy guarantee. The challenge lies in finding a reasonable and effective trade-off between model performance and privacy protection, which involves addressing three main issues: i) high privacy cost for high-dimensional data, ii) blind synthetic data generation with representation distribution of private data, and iii) priority selection to meet practical data-sensitive or label-sensitive privacy requirements. To address these challenges, this paper proposes a differentially private synthetic distillation approach to facilitate privacy-preserving model transcription (Fig. 1 ). This approach aims to convert a pretrained model (teacher) into a privacy-preserving one (student) without available private data. It takes a trainable generator as bridge and engages three players in a competitive-cooperative learning framework (Fig. 2 ), where the student collaborates with the teacher through knowledge distillation and competes with the generator via adversarial learning. Through alternate optimization of synthetic data generation and model training, the competitive-cooperative learning process achieves convergence when the student mimics the teacher in inference capability, and the generator produces synthetic data that match the representation distribution of private data. In the learning framework, the teacher and student collaborate to efficiently annotate synthetic data in a differentially private manner, facilitating the transfer of teacher knowledge to the student and effectively mitigating the issue of high privacy cost . Despite the absence of private data, the student inherits the teacher’s ability to discriminate on private data and acts as a discriminator in competition with the generator. This setup makes blind synthetic data generation effective, as the student’s refined discriminative capability guides the generation of synthetic data that closely resemble private data distribution. Further, our approach offers switchable priority selection in a unified framework to protect either data-sensitive privacy via Gaussian mechanism or label-sensitive privacy via randomized response mechanism, catering to specific privacy needs ( e.g. , in preference to protecting facial images or identities). To this end, we design theoretical differentially private annotation to balance model performance and privacy protection by i) gradient normalization in Gaussian mechanism to retain relative scale information, ii) utilizing student outputs in randomized response mechanism to retain more information and maintain privacy integrity, and iii) exploiting lower-dimensional labels to achieve differential privacy. In this way, we can theoretically validate both privacy and convergence guarantee of our approach, achieving a privacy-preserving student with minimal accuracy loss. Our major contributions are three folds: 1) we propose a differentially private synthetic distillation approach for model transcription without available private data, 2) we propose a unified framework to support two flexible priority selection in protecting data or label privacy under guarantees in theory, and 3) we conduct extensive experiments and analysis

Read the complete text