RAPID: Retrieval Augmented Training of Differentially Private Diffusion Models

Tanqiu Jiang †, Changjiang Li †, Fenglong Ma ⋆, Ting Wang †, ⋆ Pennsylvania State University · 2025

Read RAPID: Retrieval Augmented Training of Differentially Private Diffusion Models by Tanqiu Jiang †, Changjiang Li †, Fenglong Ma ⋆, Ting Wang †, ⋆ Pennsylvania State University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12794.txt.

Introduction

Tanqiu Jiang and co-authors present RAPID, a 2025 method for training image making models on private data. These models, called diffusion models, can make strong images, but training them with privacy guards often hurts quality and costs much memory and time. RAPID aims to keep the privacy guard while cutting those losses.

The method builds a store of sample paths from public data. During private training, early steps act as queries, and the model pulls like paths from the store to stand in for them. Training then focuses on the later steps under the privacy guard. In tests on standard data sets, RAPID beat past methods by wide margins on quality, memory use, and speed, with the same privacy promise.

Catalog record

CREATORTanqiu Jiang †, Changjiang Li †, Fenglong Ma ⋆, Ting Wang †, ⋆ Pennsylvania State University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12794

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RAPID: Retrieval Augmented Training of Differentially Private Diffusion Models Tanqiu Jiang †, Changjiang Li †, Fenglong Ma ⋆, Ting Wang †, ⋆ Pennsylvania State University Abstract Differentially private diffusion models (DPDMs) harness the remarkable generative capabilities of diffusion models while enforcing differential privacy (DP) for sensitive data. However, existing DPDM training approaches often suffer from significant utility loss, large memory footprint, and expensive inference cost, impeding their practical uses. To overcome such limitations, we present Rapid 1 1 1 Rapid : R etrieval- A ugmented P r I vate D iffusion model. , a novel approach that integrates retrieval augmented generation (RAG) into DPDM training. Specifically, Rapid leverages available public data to build a knowledge base of sample trajectories; when training the diffusion model on private data, Rapid computes the early sampling steps as queries, retrieves similar trajectories from the knowledge base as surrogates, and focuses on training the later sampling steps in a differentially private manner. Extensive evaluation using benchmark datasets and models demonstrates that, with the same privacy guarantee, Rapid significantly outperforms state-of-the-art approaches by large margins in generative quality, memory footprint, and inference cost, suggesting that retrieval-augmented DP training represents a promising direction for developing future privacy-preserving generative models. The code is available at: https://github.com/TanqiuJiang/RAPID . 1 Introduction The recent advances in diffusion models have led to unprecedented capabilities of generating high-quality, multi-modal data ( Ho et al., 2020 ; Kong et al., 2020 ; Bar-Tal et al., 2024 ) . However, training performant diffusion models often requires massive amounts of training data, raising severe privacy concerns in domains wherein data is sensitive. For instance, Carlini et al. (2023) show that compared with other generative models, diffusion models are especially vulnerable to membership inference attacks, due to their remarkable modeling capabilities; meanwhile, Wen et al. (2023) show that text-conditional diffusion models trained with text-image pairs can produce images almost identical to certain training samples with proper prompting. This pressing need has spurred intensive research on enforcing privacy protection in diffusion model training. Notably, Dockhorn et al. (2023) proposed the concept of differentially private diffusion models (DPDMs), which incorporate DP-SGD ( Abadi et al., 2016 ) into diffusion model training, providing guaranteed privacy; Ghalebikesabi et al. (2023) further applied DP during diffusion model fine-tuning, first pre-training a denoising diffusion probabilistic model ( Ho et al., 2020 ) on public data and then fine-tuning the model on private data under DP constraints. Similarly, Lyu et al. (2023) employed the same strategy but extended it to latent diffusion models ( Rombach et al., 2022 ) . However, existing methods suffer from major limitations. ( i ) Significant utility loss – The quality of generated samples often drops sharply under tightened privacy budgets. For instance, DpDm ( Dockhorn et al., 2023 ) fails to synthesize recognizable images on CIFAR10 under a DP budget of \epsilon=1 . ( ii ) Large memory footprint – To reduce the noise magnitude applied at each iteration, most approaches adopt excessive batch sizes (e.g., B = 8,192 samples per batch). As common DP frameworks (e.g., Opacus ( Yousefpour et al., 2021 ) ) have peak memory requirements of \mathcal{O}(B^{2}) , this severely limits the size of usable diffusion models. ( iii ) Expensive inference cost – Similar to non-private diffusion models, existing methods require expensive, iterative sampling at inference, impeding them from synthesizing massive amounts of data. To address such challenges, we present Rapid , a novel approach that integrates retrieval augmented generation (RAG) ( Lewis et al., 2020 ; Blattmann et al., 2022 ) into DPDM training. Our approach is based on the key observation that small perturbations to the early sampling steps of diffusion models have a limited impact on the overall sampling trajectory ( Khalil, 2008 ) . This allows us to reuse previously generated trajectories, if similar to the current one, as effective surrogates to reduce both training and inference costs ( Zhang et al., 2023 ) . Leveraging this idea, Rapid utilizes available public data to pre-train a diffusion model and build a knowledge base of sampling trajectories. It further refines the model using private data: it first computes the early sampling steps, retrieves similar trajectories from the knowledge base as surrogates, and focuses on training the later sampling steps under DP constraints. Compared to prior work, Rapid offers several major advantages: – It achieves a more favorable privacy-utility trade-off by fully utilizing public data and only training the later sampling steps on private data; – It significantly reduces batch-size requirements by leveraging the retrieved sample trajectories, making the use of large diffusion models feasible; – It greatly improves inference efficiency by skipping intermediate sampling steps via RAG. Extensive evaluation using benchmark datasets and models demonstrates that Rapid outperforms state-of-the-art methods by large margins in there key areas: ( i ) generative quality (e.g., improving FID score to 63.2 on CIFAR10 under a DP budget \epsilon=1 ), ( ii ) memory footprint (e.g., reducing the required batch size to just 64 samples per batch), and ( iii ) inference efficiency (e.g., saving up to 50% of the inference cost). Our findings suggest that integrating RAG into DP training represents a promising direction for developing future privacy-preserving generative models. 2 Related Work Differentially private data generation. As an important yet challenging problem, enforcing DP into training a variety of advanced generative models has attracted intensive research effort ( Hu et al., 2023 ) , including generative adversarial networks ( Goodfellow et al., 2014 ; Chen et al., 2020a ; Yoon et al., 2019 ) , variational autoencoders ( Jiang et al., 2022 ) , and customized architectures ( Liew et al., 2022 ; Vinaroz et al., 2022 ; Harder et al., 2021 ) . For instance, Harder et al. (2023) pre-train perceptual features using public data and fine-tune only data-dependent terms using maximum mean discrepancy under the DP constraint. Differentially private diffusion models. In contrast, the work on privatizing diffusion models is relatively limited. Notably, DpDm ( Dockhorn et al., 2023 ) integrate DP-SGD ( Abadi et al., 2016 ) with a score-based diffusion model ( Song et al., 2021 ) ; Ghalebikesabi et al. (2023) propose to pre-train a diffusion model with public data and then fine-tune the model using DP-SGD on private data; Dp-Ldm ( Lyu et al., 2023 ) apply a similar fine-tuning strategy to a latent diffusion model ( Rombach et al., 2022 ) ; PrivImage ( Li et al., 2024 ) queries the private data distribution to select semantically similar public samples for pretraining, followed by DP-SGD fine-tuning on the private data. However, all these methods share the drawbacks of significant utility loss, excessive batch sizes, or expensive inference costs. Beyond the pre-training/fine-tuning paradigm, recent work also explores synthesizing DP datasets by querying commercial image generation APIs (e.g., Stable Diffusion and DALL-E2) to approximate the distributions of private data ( Wang et al., 2024 ; Lin et al., 2024 ) . Retrieval augmented generation. Initially proposed to enhance the generative quality of NLP models by retrieving related information from external sources Khandelwal et al. (2019) ; Lewis et al. (2020) ; Guu et al. (2020) , RAG has been extended to utilize local cohorts in the training

Read the complete text