Mitigating data replication in text-to-audio generative diffusion models through anti-memorization guidance
Francisco Messina, Francesca Ronchini, Luca Comanducci, Paolo Bestagini, Fabio Antonacci · 2025
Read Mitigating data replication in text-to-audio generative diffusion models through anti-memorization guidance by Francisco Messina, Francesca Ronchini, Luca Comanducci, Paolo Bestagini, Fabio Antonacci (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14934.txt.
Introduction
Francisco Messina and four co-authors fight data copying in text-to-audio generation in a 2025 audio AI paper. Generative models sometimes reproduce chunks of their training data, a problem called data replication or memorization. For audio models trained on music and sound libraries, that raises legal and creative concerns.
The team applies Anti-Memorization Guidance, which steers the sampling process of a pre-trained diffusion model away from memorized content. Three guidance variants are explored, each aiming to cut replication while keeping sound quality and prompt alignment. Experiments use Stable Audio Open, chosen because both the model and its training set are fully open. Results suggest the guidance sharply reduces memorization without hurting fidelity, offering a practical safeguard for open audio generation.
Catalog record
| CREATOR | Francisco Messina, Francesca Ronchini, Luca Comanducci, Paolo Bestagini, Fabio Antonacci |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-14934
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating data replication in text-to-audio generative diffusion models through anti-memorization guidance Francisco Messina, Francesca Ronchini, Luca Comanducci, Paolo Bestagini, Fabio Antonacci Abstract A persistent challenge in generative audio models is data replication, where the model unintentionally generates parts of its training data during inference. In this work, we address this issue in text-to-audio diffusion models by exploring the use of anti-memorization strategies. We adopt Anti-Memorization Guidance (AMG), a technique that modifies the sampling process of pre-trained diffusion models to discourage memorization. Our study explores three types of guidance within AMG, each designed to reduce replication while preserving generation quality. We use Stable Audio Open as our backbone, leveraging its fully open-source architecture and training dataset. Our comprehensive experimental analysis suggests that AMG significantly mitigates memorization in diffusion-based text-to-audio generation without compromising audio fidelity or semantic alignment. † † address: Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Italy 1 Introduction Automatic audio generation has grown rapidly with the rise of multimodal text-to-audio models, which generate short audio clips from user-provided captions [ 1 , 2 , 3 ] . Their accessibility has lowered technical barriers [ 4 ] , and their commercial success in music generation has raised ethical concerns, especially around copyright and intellectual property [ 5 , 6 ] . Diffusion-based techniques [ 7 ] have rapidly become one of the most widely adopted families of generative models. However, these models are also known to exhibit memorization, i.e., reproducing content directly from the training set, creating further concerns for intellectual property rights of the training data [ 8 , 9 ] . This phenomenon has been studied extensively in the context of image generation, where duplicated images or captions in the training corpus, as well as highly specific user prompts, have been identified as primary drivers of memorization [ 10 , 11 , 12 ] . To address replication in diffusion models for image generation, several strategies have been explored [ 13 ] . One approach removes duplicated samples from the training set [ 9 ] ; while effective, it requires costly retraining. Prompt-based methods have also been investigated: randomization can reduce replication but often harms image fidelity [ 11 ] , whereas duplication suppresses certain concepts but demands substantial manual effort [ 14 ] . In the audio domain, research on memorization in audio generative models remains limited. Bralios et al. [ 8 ] conduct an initial study on data replication in the TANGO model [ 15 ] , while Batlle et al. [ 16 ] evaluate metrics for their effectiveness in detecting memorization. This paper presents the first study, to the best of our knowledge, on mitigating data replication in generative diffusion models for audio. Our approach builds on the Anti-Memorization Guidance (AMG) framework [ 12 ] , originally proposed for image generation, which operates at inference time by steering the model away from memorized samples. We focus on two primary sources of memorization: highly detailed user prompts and repeated content in the training data [ 12 ] . For our experiments, we use the Stable Audio Open model [ 17 ] , a latent diffusion transformer [ 18 ] capable of generating variable-length stereo audio. Both the model and its training dataset are fully open source, providing full transparency and control over the data used in our evaluation. The results show that AMG effectively mitigates memorization of training data, while preserving high audio quality. This confirms its ability to reduce replication without compromising the generative capabilities of the model. Audio excerpts and the code used for the experiments are freely available online 1 1 1 https://polimi-ispl.github.io/anti-memorization-tta/ . 2 Background and Preliminaries This section provides a concise overview of the background necessary to facilitate a clear understanding of the study. 2.1 Latent Diffusion Models Let us define \mathbf{x}\in\mathbb{R}^{N} as a discrete audio signal of length N samples. We consider an encoder \mathcal{E} and a decoder \mathcal{D} , such that we can obtain the latent representation \mathbf{z}=\mathcal{E}(\mathbf{x}) , from which we then retrieve the audio as \mathbf{x}=\mathcal{D}(\mathbf{z}) . The latent diffusion model [ 19 ] , is then characterized by a forward process, which gradually adds gaussian noise to a latent vector \mathbf{z}_{0}\sim q(\mathbf{z}) using T timesteps, such that \mathbf{z}_{T}\sim\mathcal{N}(0,\mathbf{I}) . Then it is possible to sample \mathbf{z}_{t} at an arbitrary timestep t as q(\mathbf{z}_{t}|\mathbf{z}_{0})=\mathcal{N}(\mathbf{z}_{t},\sqrt{\overline{\alpha}_{t}}\mathbf{z}_{0},(1-\overline{\alpha}_{t}\mathbf{I})), (1) where \overline{\alpha}_{t}=\prod_{s=0}^{t-1}(1-\beta_{s}) and \beta_{t}\in\mathbb{R},\;t=0,\ldots,T-1 is the noise schedule controlling the amount of noise added at each timestep. Then, in the backward process, at an arbitrary step t the noise \boldsymbol{\epsilon}_{t} is predicted via network \epsilon_{\theta} by minimizing the loss \mathcal{L}=\mathbb{E}_{\boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I}),t}[||\boldsymbol{\epsilon}_{t}-\epsilon_{\theta}(\mathbf{z}_{t},t)||_{2}^{2}], (2) where the conditioning can be simply inserted by adding a label to the argument of \epsilon_{\theta}(\cdot) . 2.2 Classifier-Free Guidance Classifier-Free Guidance (CFG) is a technique that enables efficient conditioning by training at the same time over conditional and unconditional objectives [ 20 ] , by simply using an empty label during the unconditional case. During inference, the prediction is steered towards the conditional one using \hat{\boldsymbol{\epsilon}}\longleftarrow\epsilon_{\theta}(\mathbf{z}_{t})+s_{0}+(\epsilon_{\theta}(\mathbf{z_{t}},y)-\epsilon_{\theta}(\mathbf{z}_{t})), (3) where s_{0}>1 controls the adjustment level and y is the conditioning. Please note that the inference mechanism of CFG operates in a way that is conceptually similar to the mitigation strategies discussed in the remainder of this paper. (a) (b) (c) Figure 1: Example spectrograms of original training audio track (a) and generated using the same textual prompt without (b) and with AMG (c). 3 Method This section introduces the Anti-Memorization Guidance (AMG) framework [ 12 ] , adapted for audio latent diffusion models. 3.1 Anti-memorization guidance framework The AMG framework continuously monitors the generation process, intervening only when the model’s output closely matches a known training example. To formalize this, let \hat{\mathbf{x}}\in\mathbb{R}^{N} denote audio generated by a diffusion model, and let \mathbf{x}\in\mathbb{R}^{N} represent an audio sample from the training dataset \mathcal{X} . The most similar training example can be identified by finding its nearest neighbor \boldsymbol{\nu} , computed as the sample in \mathcal{X} with the smallest \ell_{2} distance to \hat{\mathbf{x}} . Since computing the \ell_{2} norm between the 1D waveforms is highly sensitive to time shifts and other variations, we instead compute the distance between descriptors extracted via a network. Specifically, CLAP laion [ 21 ] , following the approach of [ 8 ] , and denote the CLAP embedding function as f_{\text{CLAP}}(\cdot) . The nearest neighbor in descriptor space is then defined as: \boldsymbol{\nu}=\arg\min_{\mathbf{x}\in\mathcal{X}}\;\|f_{\text{CLAP}}(\hat{\mathbf{x}})-f_{\text{CLAP}}(\mathbf{x})\|_{2}. (4) We then define a similarity measure to quantify the degree of memorization for an individual track, based on the cosine distance between the generated track and its nearest neighbor: \sigma_{t}=\frac{f_{\text{CLAP}}(\hat{\mathbf{x}})\cdot f_{\text{CLAP}}(\mathbf{x})}{\lVert f_{\text{CLAP}}(\hat{\mathbf{x}})\rVert\,\lVert f_{\text{CLAP}}(\mathbf{x})\rVert}. (5) The AMG framework introduces three complementary guidance strategies, each targeting a distinct source of memorization, in the form of vectors summed to the noise in order to steer the prediction away from the memorized data. Specifically, it includes: despecification guidance \mathbf{g}_{\text{spe}} , which mitigates the influence of the user’s prompt; caption deduplication guidance \mathbf{g}_{\text{dup}} , which uses a memorized example’s caption as a negative prompt; and dissimilarity guidance \mathbf{g}_{\text{sim}} , which actively steers the generation away from its nearest neighbor in the embedding space. AMG operates exclusively at inference time, applied directly during the reverse process. Concretely, the noise \boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) is updated at each inference step as: \hat{\boldsymbol{\epsilon}}\longleftarrow\hat{\boldsymbol{\epsilon}}+\mathbbm{1}_{\{\sigma_{t}>\lambda_{t}\}}\bigl(\mathbf{g}_{\text{spe}}+\mathbf{g}_{\text{dup}}+\mathbf{g}_{\text{sim}}\bigr), (6) where