Memorization Control in Diffusion Models from Denoising-centric Perspective
Thuy Phuong VuDepartment of Computer ScienceBrunel University of LondonLondon, United KingdomMai Viet Hoang DoHanoi University of Science and TechnologyHanoi, VietnamMinhhuy LeFaculty of Electrical and Electronic EngineeringSchool of EngineeringPhenikaa UniversityHanoi, VietnamDinh-Cuong HoangGreenw · 2026
Read Memorization Control in Diffusion Models from Denoising-centric Perspective by Thuy Phuong VuDepartment of Computer ScienceBrunel University of LondonLondon, United KingdomMai Viet Hoang DoHanoi University of Science and TechnologyHanoi, VietnamMinhhuy LeFaculty of Electrical and Electronic EngineeringSchool of EngineeringPhenikaa UniversityHanoi, VietnamDinh-Cuong HoangGreenw (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21348.txt.
Introduction
Thuy Phuong Vu and co-authors study memorization in diffusion models from a fresh angle. These models can copy training images too closely, which is a problem when new output should match the data's pattern without cloning it. Most fixes change the data or the model itself.
The authors instead look at the denoising process, the step-by-step cleanup that builds each image. Viewing the model as part of a wider pipeline changes where control can be applied. Their angle suggests new ways to dial memorization up or down. That control matters for privacy and for fair use of training data.
Catalog record
| CREATOR | Thuy Phuong VuDepartment of Computer ScienceBrunel University of LondonLondon, United KingdomMai Viet Hoang DoHanoi University of Science and TechnologyHanoi, VietnamMinhhuy LeFaculty of Electrical and Electronic EngineeringSchool of EngineeringPhenikaa UniversityHanoi, VietnamDinh-Cuong HoangGreenw |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21348
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Memorization Control in Diffusion Models from Denoising-centric Perspective Thuy Phuong VuDepartment of Computer ScienceBrunel University of LondonLondon, United KingdomMai Viet Hoang DoHanoi University of Science and TechnologyHanoi, VietnamMinhhuy LeFaculty of Electrical and Electronic EngineeringSchool of EngineeringPhenikaa UniversityHanoi, VietnamDinh-Cuong HoangGreenwich VietnamFPT UniversityHanoi, Vietnam, Phan Xuan Tan College of EngineeringShibaura Institute of TechnologyTokyo, Japan Abstract Controlling memorization in diffusion models is critical for applications that require generated data to closely match the training distribution. Existing approaches mainly focus on data-centric or model-centric modifications, treating the diffusion model as an isolated predictor. In this paper, we study memorization in diffusion models from a denoising-centric perspective. We show that uniform timestep sampling leads to unequal learning contributions across denoising steps due to differences in signal-to-noise ratio, which biases training toward memorization. To address this, we propose a timestep sampling strategy that explicitly controls where learning occurs along the denoising trajectory. By adjusting the width of the confidence interval, our method provides direct control over the memorization–generalization trade-off. Experiments on image and 1D signal generation tasks demonstrate that shifting learning emphasis toward later denoising steps consistently reduces memorization and improves distributional alignment with training data, validating the generality and effectiveness of our approach. 1 Introduction Diffusion models (DMs) [ 1 ] have been widely adopted across a broad range of applications, extending beyond image generation and image editing to domain-specific tasks such as data augmentation in scientific and industrial settings [ 2 ] . Unlike creative image generation, where diversity and novelty are highly desired, many domain-specific applications require generated samples to strictly follow the training data distribution, rendering excessive creativity unnecessary or even harmful. In such scenarios, the ability to control memorization in diffusion models becomes critically important. Recent studies have established, both empirically and theoretically, that memorization can indeed arise in diffusion models [ 3 , 4 ] . In particular, [ 5 ] demonstrate that the sampling process of diffusion models can be partitioned into generalization and memorization regimes. During early denoising steps, when noise levels are high, the generated samples exhibit more general, distribution-level characteristics, whereas later steps closer to the final output introduce increasingly specific and detailed structures. Moreover, the authors show that the sampling trajectory naturally drifts into an extrapolation regime. As a consequence, deviations introduced during early denoising steps when compounded by the largely deterministic nature of the denoising process can prevent the generated samples from converging back to the training data distribution. While prior work has explored controlling memorization through data manipulation, model capacity, architectural design, or loss-based regularization, such approaches largely treat diffusion models as isolated predictors. In contrast, diffusion models fundamentally operate as components within a sequential denoising process, where model predictions are repeatedly applied and accumulated over time. Therefore, adjusting memorization behavior in diffusion models requires modifying not only what the model learns, but also how it learns within the denoising dynamics. Motivated by this perspective, we approach memorization control from a denoising-centric viewpoint, explicitly accounting for the role of the model as an integral part of the denoising process. In this paper, we make the following contributions. First, we provide a theoretical analysis of diffusion model training through denoising perspective, showing that despite uniform timestep sampling, different denoising steps contribute unequally to the optimization objective. We formally demonstrate that this imbalance leads to biased denoising behavior, which in turn contributes to the emergence of memorization. From this observation, we propose a denoising-centric training strategy that explicitly adjusts the model’s learning behavior. Rather than encouraging uniform learning across both generalization and memorization regimes, we design the model to focus on learning effectively in the more challenging regime, thereby preventing denoising trajectories from drifting into extrapolation regions. Finally, we evaluate our hypothesis on both image generation and 1D data generation tasks, demonstrating that our approach is effective not only for highly creative tasks such as 2D image synthesis, but also for domain-specific scenarios where controlled memorization is required. 2 Related works Diffusion models learn a score function from noisy data and use it during sampling to generate images [ 1 ] . Memorization and generalization in DMs exist and the nature of the transition from memorization to generalization are mathematically shown in research of [ 3 , 4 ] . If the model were to learn the exact score of the training data everywhere in the input space, it would merely reproduce the training samples and fail to generate novel data. Instead, the model underfits only in certain regions of the input space while fitting well in others, a phenomenon termed selective underfitting [ 5 ] , which challenges the view that diffusion models underfit the entire score function. The authors distinguish between a supervision region and an extrapolation region, but do not provide a concrete mechanism to explicitly control the trade-off between these two regions. To address memorization, [ 6 ] introduce a framework that adjusts model outputs during sampling to prevent memorization while preserving generation quality. Unlike prior studies focusing on training dynamics or model architecture, it operates at inference time, selectively intervening step-by-step only when memorization is detected, thus minimizing disruption to the standard sampling process. However, since this approach intervenes only at the sampling stage and primarily filters or redirects outputs, it does not address the root causes of memorization, making the mitigation inherently dependent on stochastic sampling behavior. Recently, many works impress on the importance of data in their memorization. [ 4 ] highlights the critical role of data geometry in memorization, mathematically showing that diffusion models’ tendency to memorize is strongly influenced by the manifold structure of the training data. [ 7 ] attributes the lack of memorization in diffusion models to implicit dynamical regularization during training, with the generalization–memorization timescale separation depending on dataset size. Additionally, [ 8 ] states that generalization and memorization are mutually exclusive phenomena and by reducing model capacity and complexity of train data as injecting dummy data, it can be prevented and increase generalization. Moreover, memorization is also affected by time embedding method or some training configurations as training epochs, batch size, weight decay [ 9 ] . [ 10 ] also state that transition from generalization to memorization is governed by the loss landscape and depends on both data complexity and model capacity. They identify a crossover point between them, determined by the intersection of training losses for generalizing and memorizing denoisers. However, [ 11 ] explains memorization through a provable separation between empirical and ground-truth objectives and mitigates it via model-capacity control rather than loss optimization. While prior work studies memorization from data, model, or training perspectives, it largely overlooks the multi-step denoising nature of diffusion models, in which memorization may emerge as learned biases are repeatedly carried over and strengthened across denoising steps. Unlike earlier works that attribute memorization to data, training timescales, or loss landscape transitions, [ 12 ] emphasizes the role of data–model interactions during denoising. The authors explain memorization as a consequence of biased early denoising predictions amplified by classifier-free guidance. However, early timesteps receive limited training signal due to weak gradient contributions, raising the question of whether these early predictions are sufficiently informative to meaningfully guide the sampling process. Motivated by these observations, we control memorization