Filtering Memorization from Parameter-Space in Diffusion Models
Yu Zhe † \dagger, Yang Jiayan † \dagger, Wei Junhao, Yu-Lin Tsai, Wang Chen · 2026
Read Filtering Memorization from Parameter-Space in Diffusion Models by Yu Zhe † \dagger, Yang Jiayan † \dagger, Wei Junhao, Yu-Lin Tsai, Wang Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10439.txt.
Introduction
Yu Zhe and four colleagues address a copyright risk hiding inside shared AI customizations. Low-Rank Adaptation, or LoRA, lets users inject new visual concepts or styles into diffusion models through light weight updates. Platforms hold huge libraries of user-uploaded LoRAs. Yet LoRAs can memorize training images, reproducing copyrighted or sensitive content. Their 2026 paper proposes Base-Anchored Filtering, or BAF, a fix needing neither training data nor pipeline access.
BAF sorts LoRA updates into spectral channels and checks how far they line up with the main subspace of the pretrained backbone. Channels strongly aligned with that subspace are kept as generalizable adaptations. Weakly aligned channels are damped down as likely carriers of memorized content. Tests across multiple datasets and diffusion backbones show BAF steadily cuts memorization while keeping or even improving generation quality, giving hosts and users a post-hoc tool for deployed LoRAs.
Catalog record
| CREATOR | Yu Zhe † \dagger, Yang Jiayan † \dagger, Wei Junhao, Yu-Lin Tsai, Wang Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10439
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Filtering Memorization from Parameter-Space in Diffusion Models Yu Zhe † \dagger, Yang Jiayan † \dagger, Wei Junhao, Yu-Lin Tsai, Wang Chen Abstract Low-Rank Adaptation (LoRA) has become a widely used mechanism for customizing diffusion models, enabling users to inject new visual concepts or styles through lightweight parameter updates. However, LoRAs can memorize training images, causing generated outputs to reproduce copyrighted or sensitive content. This risk is particularly concerning in LoRA-sharing ecosystems, where users distribute trained LoRAs without releasing the underlying training data. Existing approaches for mitigating memorization rely on access to the training pipeline, training data, or control over the inference process, making them difficult to apply when only the released LoRA weights are available. We propose Base-Anchored Filtering (BAF) , a training-free and data-free framework for post-hoc memorization mitigation in diffusion LoRAs. BAF decomposes LoRA updates into spectral channels and measures their alignment with the principal subspace of the pretrained backbone. Channels strongly aligned with this subspace are retained as generalizable adaptations, while weakly aligned channels are suppressed as potential carriers of memorized content. Experiments on multiple datasets and diffusion backbones demonstrate that BAF consistently reduces memorization while preserving or even improving generation quality. Our code is available in the supplementary material. K eywords Diffusion Models ⋅ \cdot Memorization ⋅ \cdot Low-Rank Adaptation 1 1 footnotetext: \dagger Equal contribution. 1 Introduction Text-to-image diffusion models [ 11 , 24 ] have become the dominant paradigm for high-fidelity image generation. Building on pretrained backbones such as Stable Diffusion [ 24 ] , users increasingly customize generation toward specific styles, characters, or visual concepts. Low-Rank Adaptation (LoRA) [ 12 ] enables such customization by fine-tuning a small set of low-rank weight updates from only a few reference images. This simplicity has given rise to a large-scale open ecosystem. Platforms such as Hugging Face and CivitAI host hundreds of thousands of user-uploaded LoRAs * * * CivitAI: https://civitai.com ; Hugging Face: https://huggingface.co/ , many of which are publicly available, with direct plug-and-play usage on popular diffusion backbones. While this open ecosystem enables creative flexibility, it also raises concerns regarding memorization and copyright. Particularly, memorization refers to the phenomenon whereby a model reproduces training examples or near-identical variations under certain prompts [ 2 , 28 , 31 ] . When the memorized content corresponds to copyrighted artwork, protected characters, or identifiable individuals, generation may constitute unauthorized reproduction. The risk is particularly acute in LoRA-sharing scenarios: downstream users typically download and deploy LoRAs without access to the fine-tuning data or optimization procedure, yet memorized content embedded in the weights can surface during normal generation (as shown in Fig. 1 ). Consequently, mitigating such risks becomes important not only for platforms that host and distribute LoRA models, but also for users who wish to avoid unintended copyright or privacy violations when deploying these models. Figure 1: LoRA weight carries memorization of training data, manifested as pixel-level and object-level reproduction of training samples. BAF successfully modifies diffusion models to produce memorization-free outputs. On the other hand, existing memorization mitigation strategies are largely inapplicable in this setting. Training-time interventions [ 28 , 30 ] aim to prevent memorization during the training process. Yet hosting platforms and LoRA users merely receive uploaded LoRAs and do not control the fine-tuning process. Alternatively, output-based auditing methods detect memorization by comparing generated samples against known training images [ 2 , 31 ] , while concept-erasure approaches remove specified learned concepts directly from model weights [ 7 , 8 ] . Both require prior knowledge of the memorized content, which is unavailable when only the LoRA weights are uploaded. Inference-time defenses modify the sampling process to suppress memorized outputs [ 5 ] . Such approaches can be applied by users who control the inference pipeline. However, they cannot be reliably enforced at the platform level, since hosting platforms typically only distribute LoRA weights and cannot regulate downstream generation. Consequently, these approaches rely on information or control that is unavailable in LoRA-sharing ecosystems, where only the released LoRA parameters are observable. This motivates a different formulation: how can memorization be identified and filtered when only the LoRA weights are available? To address this question, we leverage the geometric structure of the pretrained backbone as a structural anchor. Large-scale pretraining induces a highly organized spectral structure in the network parameters [ 16 , 27 ] , and meaningful task-specific variations tend to reside within low-dimensional subspaces aligned with the principal directions of the pretrained weights [ 1 ] . This discovery has led to a key insight. Generalizable adaptation follows dominant spectral directions, whereas memorization introduces weakly aligned instance-specific directions. Empirically, we decompose LoRA updates via singular value decomposition (SVD) and measure the alignment of spectral channels with the pretrained model’s principal subspace. We then observe that generation from high-alignment channels preserves the intended adaptation while exhibiting minimal memorization, whereas generation from low-alignment channels reproduces memorized training images at significantly elevated rates. This separation suggests that memorization is predominantly concentrated in spectral directions that are weakly aligned with the pretrained backbone, leaving a detectable structural signature in parameter space. Building on this observation, we introduce BAF (Base-Anchored Filtering), a training-free and data-free framework for post-hoc memorization mitigation in diffusion LoRAs. BAF decomposes each LoRA update into spectral channels via SVD and measures their alignment with the principal subspace of the pretrained backbone. Channels with strong alignment are retained as generalizable adaptations, whereas weakly aligned channels are down-weighted or removed as potential memorization carriers. By operating directly in parameter space, BAF mitigates memorization without requiring access to fine-tuning data, knowledge of memorized content, or additional optimization, making it applicable to both hosting platforms and downstream LoRA users. Our contributions are as follows: 1. We formulate the problem of post-hoc memorization mitigation for diffusion LoRAs under strictly data-inaccessible constraints. 2. We propose and validate a geometric insight distinguishing generalizable adaptation from memorization via spectral alignment with the pretrained principal subspace. 3. We introduce Base-Anchored Filtering (BAF), a training-free and data-free framework that selectively suppresses memorization-related spectral directions in parameter space. 4. We demonstrate that BAF effectively reduces memorization while preserving benign generative capability, achieving better trade-offs compared to existing memorization suppression strategies. 2 Related work 2.1 Memorization in diffusion model Memorization in diffusion models refers to the tendency of a model to reproduce training images or near-identical variants during generation. Early studies [ 28 , 2 ] showed that text-to-image systems such as Stable Diffusion replicate a non-trivial portion of their training data. Subsequent work investigated the mechanisms behind memorization. [ 9 ] analyzed the effects of dataset size, duplication rate, and model capacity, and observed that in LoRA fine-tuning, memorization strength increases with the rank of the adaptation matrices. At a different granularity, [ 31 ] linked memorization to abnormal text-conditional prediction magnitudes and proposed token-level attribution to identify triggering words. [ 23 ] localized memorization effects to cross-attention patterns, while [ 4 ] distinguished between global and local memorization, showing that the latter is harder to detect and mitigate. From a geometric perspective, [ 25 ] connected memorization to regions of near-zero Local Intrinsic Dimensionality (LID). 2.2 Memorization Detection and Mitigation Existing work on mitigating memorization can be broadly categorized into training-time, inference-time, and parameter-space