Rate of Model Collapse in Recursive Training
Ananda Theertha Suresh, Andrew Thangaraj, Aditya Nanda Kishore Khandavally · 2024
Read Rate of Model Collapse in Recursive Training by Ananda Theertha Suresh, Andrew Thangaraj, Aditya Nanda Kishore Khandavally (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17646.txt.
Introduction
How fast models collapse when trained on their own output is timed by Ananda Theertha Suresh, Andrew Thangaraj, and Aditya Nanda Kishore Khandavally in a 2024 arXiv paper. Round after round of self-made data can blur the marks of real human data. The team asks for exact speeds in two base cases: word counts and bell curves. Both were open questions till now.
For word counts, time to forget a word runs with how often it first showed. For bell curves, spread shrinks to zero at a pace set by sample size per round. Collapse, in these clean cases, is slow.
Catalog record
| CREATOR | Ananda Theertha Suresh, Andrew Thangaraj, Aditya Nanda Kishore Khandavally |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17646
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rate of Model Collapse in Recursive Training Ananda Theertha Suresh, Andrew Thangaraj, Aditya Nanda Kishore Khandavally Abstract Given the ease of creating synthetic data from machine learning models, new models can be potentially trained on synthetic data generated by previous models. This recursive training process raises concerns about the long-term impact on model quality. As models are recursively trained on generated data from previous rounds, their ability to capture the nuances of the original human-generated data may degrade. This is often referred to as model collapse . In this work, we ask how fast model collapse occurs for some well-studied distribution families under maximum likelihood (ML or near ML) estimation during recursive training. Surprisingly, even for fundamental distributions such as discrete and Gaussian distributions, the exact rate of model collapse is unknown. In this work, we theoretically characterize the rate of collapse in these fundamental settings and complement it with experimental evaluations. Our results show that for discrete distributions, the time to forget a word is approximately linearly dependent on the number of times it occurred in the original corpus, and for Gaussian models, the standard deviation reduces to zero roughly at n n iterations, where n n is the number of samples at each iteration. Both of these findings imply that model forgetting, at least in these simple distributions under near ML estimation with many samples, takes a long time. 1 Introduction Machine learning models are trained using datasets that, in most cases, are originally human generated. However, with the recent advance of generative models ( Brown et al., 2020 ; Chowdhery et al., 2023 ; Thoppilan et al., 2022 ; Touvron et al., 2023 ) , it is possible today for machine learning models to generate human-like data. This has led to the proliferation of machine-generated text and images in all forms of digital content. Hence, in the next iteration of model training, if one collects a large corpus of data such as text or images, a part of it would be human generated and a part would be machine generated. Suppose we repeat this recursive training process several times, i.e. we repeatedly train models on data generated from previous iteration of models. Then, it is reasonable to suspect that the model quality might degrade over time. Motivated by this concern, Shumailov et al. (2023) ; Shumailov et al. (2024) studied the problem of recursive training and showed that the tail of the original dataset disappears over multiple rounds of recursive training. In this work, we take a step further and ask how fast does the model quality degrade in recursive training? In order to answer this question, we study the simplified recursive training procedure of Shumailov et al. (2023) where models at current iteration are trained solely based on data generated by models at the previous iteration. Figure 1 demonstrates the iterative training procedure. Figure 1 : Simplified recursive training procedure of Shumailov et al. (2023) . The underlying dataset is denoted by S_{0} . The model parameter \Theta_{k} is trained based on dataset S_{k} and sample S_{k+1} is obtained by sampling from model with parameter \Theta_{k} . Let P_{\theta} denote a distribution over a domain {\mathcal{X}} parameterized by a vector \theta=(\theta_{1},\theta_{2},\ldots,\theta_{m}) . For example, in the case of Gaussian distributions, \theta=(\mu,\sigma^{2}) . Let S_{0}=(X_{0,1},X_{0,2},\ldots,X_{0,n}) denote the original dataset of n samples, which can come from any arbitrary distribution. Let \Theta_{k}=(\Theta_{k,1},\Theta_{k,2},\ldots,\Theta_{k,m}) denote the vector of model parameters at iteration k , and let P_{\Theta_{k}} denote the corresponding sampling distribution. For each k\geq 0 , the model P_{\Theta_{k}} is trained based on dataset S_{k}=(X_{k,1},X_{k,2},\ldots,X_{k,n}) , and then P_{\Theta_{k}} is sampled to create a dataset S_{k+1}=(X_{k+1,1},X_{k+1,2},\ldots,X_{k+1,n}) for the next iteration. We have the recursion for k=0,1,\ldots : \displaystyle\Theta_{k} \displaystyle=\widehat{\theta}(X_{k,1},\ldots,X_{k,n})\quad\text{(train)}, (1) \displaystyle S_{k{+}1}{=}( \displaystyle X_{k{+}1,1},\ldots,X_{k{+}1,n})\sim\text{iid }P_{\Theta_{k}}\ \text{(sample)}, where \widehat{\theta} denotes an estimator for the parameter vector from n iid samples in each iteration. Our goal is to study the behaviour of the random process \{\Theta_{k}:k=0,1,2,\ldots\} , and observe convergence properties of the parameters in \Theta_{k} as k\to\infty . If model quality is not degraded over recursive training, sample paths of \Theta_{k} will be ergodic and cover the entire space of possible parameters. However, if any of the parameters are tending to zero or some other trivial value, we have an instance of so-called model collapse in recursive training ( Shumailov et al., 2023 ) . In such cases, we attempt to characterise the rate (as a function of k ) at which the collapsing parameter tends to the trivial value. As an illustrative example, we plot 25 different trajectories of recursive training starting with N(0,1) in Figure 2 (a). (a) Gaussian model ( \mu_{0}=0 , \sigma_{0}=1 ) (b) Bernoulli model ( p_{0}=0.2 ) Figure 2 : Multiple trajectories of recursive training across generations in the Gaussian and Bernoulli models, with n=100 samples per generation. Observe that the standard deviation tends to zero always and the means tend to different values. Figure 2 (b) 1 1 1 Code can be found at www.github.com/berserank/rate-of-model-collapse/ shows 25 trajectories for recursive training of Bernoulli samples. When started with Bernoulli (0.2) , the estimated Bernoulli parameter tends to zero or one in each trajectory. The recursive training iteration in equation 1 can be termed a stochastic recursion, which is a broad topic with applications ranging from dynamical systems and optimization to reinforcement learning ( Borkar, 2022 ; Diaconis and Freedman, 1999 ) (see Appendix A for more details). While the nature of the stochastic recursion in equation 1 does not match exactly with previously studied cases, a lot of underlying ideas are useful. Firstly, there is a Markov structure inherent in equation 1 . Given S_{k} , the future models do not depend on the S_{k-1} and the past. Secondly, under suitable unbiasedness conditions on \widehat{\theta} , we can expect martingale properties ( Williams, 1991 ) to hold for certain functionals of \Theta_{k} . Finally, for convergence in stochastic recursions, an average contraction property in each step of the recursion has been seen to be very useful to prove convergence bounds. So, one viable approach is to prove contraction properties for the expectation of a suitable function of parameters in \Theta_{k} and ultimately derive convergence rates. 1.1 Our contributions In this article, we primarily consider two types of models. Motivated by the language modelling applications, we first study learning a discrete distribution P_{\theta} with probability mass function \theta=(\theta_{1},\ldots,\theta_{m}) over a finite alphabet \{1,2,\ldots,m\} . In the terminology of language models, \theta corresponds to the unigram distribution. Under the iterations in equation 1 , if \Theta_{k,i}\to 0 as k\to\infty , the model forgets the symbol i under recursive training, and this is an instance of model collapse. Secondly, motivated by image generation tasks, we study one-dimensional Gaussian and Gaussian mixture distributions where \theta=(\mu,\sigma) . In the Gaussian case, we have P_{\theta}=N(\mu,\sigma^{2}) , which denotes the Gaussian distribution with mean \mu and variance \sigma^{2} . In the Gaussian mixture case, we let P_{\theta}=\frac{1}{2}N(-\mu,\sigma^{2})+\frac{1}{2}N(\mu,\sigma^{2}) . In these two cases, if the variance parameter tends to zero under recursive training, we have model collapse. During our analysis, we also characterize rates to model collapse for several models such as Bernoulli and Poisson models, which might be interesting on their