Model non-collapse: Minimax bounds for recursive discrete distribution estimation
Millen Kanabar, Michael Gastpar · 2025
Read Model non-collapse: Minimax bounds for recursive discrete distribution estimation by Millen Kanabar, Michael Gastpar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19273.txt.
Introduction
Millen Kanabar and Michael Gastpar ask when learning from your own output poisons the well, and when it does not, in a 2025 theory paper. Generative models are increasingly trained on data mixed with their own past output, a loop called self-consumption. In fully synthetic loops, models are seen to collapse generation by generation.
Adding fresh real data seems to prevent full collapse, and the authors formalize that setting: an estimator is fixed, its samples join the corpus, and the next estimator trains on the blend. They derive minimax bounds, the best achievable error, for this recursive estimation. The bounds mark how much real data inflow keeps the loop healthy. The result gives machine learning practice a firmer footing for an already common habit.
Catalog record
| CREATOR | Millen Kanabar, Michael Gastpar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Model non-collapse: Minimax bounds for recursive discrete distribution estimation Thanks: The work in this paper was supported in part by the Swiss National Science Foundation under Grant 200364. Thanks: Email: [email protected] and [email protected] Millen Kanabar, Michael Gastpar Abstract Learning discrete distributions from i.i.d. samples is a well-understood problem. However, advances in generative machine learning prompt an interesting new, non-i.i.d. setting: after receiving a certain number of samples, an estimated distribution is fixed, and samples from this estimate are drawn and introduced into the sample corpus, undifferentiated from real samples. Subsequent generations of estimators now face contaminated environments, a scenario referred to in the machine learning literature as self-consumption. Empirically, it has been observed that models in fully synthetic self-consuming loops collapse—their performance deteriorates with each batch of training—but accumulating data has been shown to prevent complete degeneration. This, in turn, begs the question: What happens when fresh real samples are added at every stage? In this paper, we study the minimax loss of self-consuming discrete distribution estimation in such loops. We show that even when model collapse is consciously averted, the ratios between the minimax losses with and without source information can grow unbounded as the batch size increases. In the data accumulation setting, where all batches of samples are available for estimation, we provide minimax lower bounds and upper bounds that are order-optimal under mild conditions for the expected ℓ 2 2 \ell_{2}^{2} and ℓ 1 \ell_{1} losses at every stage. We provide conditions for regimes where there is a strict gap in the convergence rates compared to the corresponding oracle-assisted minimax loss where real and synthetic samples are differentiated, and provide examples where this gap is easily observed. We also provide a lower bound on the minimax loss in the data replacement setting, where only the latest batch of samples is available, and use it to find a lower bound for the worst-case loss for bounded estimate trajectories. I Introduction The problem of estimating distributions from their samples arises naturally in various contexts, and has been studied in depth in the literature. The exact minimax loss of discrete distribution estimation under the \ell_{2}^{2} metric, among others, was found in [ 1 ] , and was studied under \ell_{1} metric e.g. in [ 2 , 3 , 1 ] . Evidently, the results of these estimations are often themselves valid distributions. These estimates can then themselves be used to generate new samples in the same alphabet. In cases where distribution estimation is performed in batches, samples generated from estimates at some stage can be reintroduced into subsequent batches of ‘real’ samples, thereby affecting future performance. This phenomenon has been widely observed, for example, in the context of generative machine learning, where distributions are estimated primarily to produce new samples, in part because it is significantly cheaper to generate synthetic samples. This has also led to fears that synthetic samples might significantly contaminate real databases such as the Internet [ 4 ] . It has been observed empirically e.g. in [ 5 , 6 ] that this process, without external mitigation, leads to ‘model collapse’: worsening performance with every iteration where samples from previous stages are introduced into the training pool. The self-referential nature of the sampling process causes the distributions of later samples to depend non-trivially on the previous batches, and thus prevents the direct use of already existing results from the distribution estimation literature. In this paper, we study the properties of such sequences of estimates under a minimax lens. A sketch of the data production workflow considered in our paper is shown in Figure 1 . A crucial aspect of the model is that the corpus does not include the identities of the source of the samples, i.e. whether they are real or synthetic. In the data accumulation setting, new samples are continually added to the corpus at every stage, whereas it consists of only fresh samples in the data replacement setting. This mirrors quite a few variants considered in the generative machine learning literature. In this paper, we assume that each sample is drawn from either the true underlying distribution or the previous estimate based on the outcome of a coin flip, independent of every other sample, with stage-dependent bias \alpha_{t} which is assumed to be known to the estimator. Fig. 1: The estimation workflow in the data accumulation setting; solid arrows indicate sampling (with the corresponding sampling probability indicated next to them), whereas dotted arrows indicate estimation. Under the data replacement setting, the corpus used for estimation only contains the latest batch of samples. I-A Contributions In this work, we pose the multi-stage distribution estimation problem with self-consumption of samples, which has not been studied in the minimax setting before in the literature. We analyze the minimax loss in terms of the sequences n_{t} , denoting the total number of samples in batch t , and \alpha_{t} , the probability with which each sample in batch t is drawn from the real distribution. We consider the oracle-assisted minimax loss—where the source distribution of each sample is provided to the estimator by an oracle—as a baseline. It is perhaps unsurprising that the oracle-assisted minimax loss at stage t is \Theta(\nicefrac{{1}}{{\sum_{i=0}^{t}n_{i}\alpha_{i}}}) , since the expected number of real samples seen after t batches is precisely \sum_{i=0}^{t}n_{i}\alpha_{i} . We provide the first (to the best of our knowledge) information-theoretic lower bounds on the \ell_{2}^{2} and \ell_{1} losses at each stage in terms of properties of the estimates at previous stages in Theorem 1 (for the data accumulation workflow) and Corollary 4 (for the data replacement workflow). We also propose a sequence of estimators that is order-optimal under certain regimes with stage-wise performance guarantees in Theorem 2 . We show conditions where there are gaps between the upper bound and lower bounds on the minimax loss and the oracle-assisted minimax loss the data accumulation workflow (Propositions 2 and 3 ) and provide examples where these conditions are satisfied. We briefly discuss the performance of estimators that do not account for the introduction of synthetic samples and show that results shown in the Model Collapse literature, where collapse is avoided by simply following the accumulate workflow still hold in our setting (Corollary 3 ). Finally, we use our lower bound for the data replacement workflow to find a time-independent lower bound on the worst-case (in ground truth distribution) loss given a time-independent upper bound. We find that the minimax loss stays within a constant factor of the oracle-assisted one when \alpha_{t} is lower bounded by a constant. When \alpha_{t} decays to 0 , i.e. when synthetic sampling outpaces real sampling, the effective sample size of batch t is proportional to n_{t}\alpha_{t}^{2} (as opposed to simply n_{t}\alpha_{t} for the oracle-assisted setting) in most practically conceivable conditions. As a consequence, the ratio of the minimax loss to the oracle-assisted minimax loss can grow unbounded with t (e.g. Claim 1 ). We specify the conditions for this regime in Proposition 2 ; the basic requirement is that \alpha_{t} does not decay too quickly. When \alpha_{t} does decay quickly, an error term starts dominating the lower bound, and there is a gap between the upper and lower bounds. However, under mild conditions, there is