ForTIFAI: Fending Off Recursive Training Induced Failure for AI Models

Soheil Zibakhsh Shabgahi, Pedram Aghazadeh, Azalia Mirhosseini, Farinaz Koushanfar · 2025

Read ForTIFAI: Fending Off Recursive Training Induced Failure for AI Models by Soheil Zibakhsh Shabgahi, Pedram Aghazadeh, Azalia Mirhosseini, Farinaz Koushanfar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08972.txt.

Introduction

Soheil Zibakhsh Shabgahi and 3 co-authors propose ForTIFAI to fight model collapse from recursive training in a 2025 paper. As synthetic data grows, models trained again and again on it degrade over generations. Effective defenses against that collapse remain scarce.

Autoregressive models tend to give high confidence to their own generated text. A new loss, Truncated Cross Entropy, ignores high confidence tokens during training, filtering likely machine artifacts. Models trained this way tolerate more than twice as much synthetic data before collapse begins. An open benchmark for collapse in mixed data is also provided. The tool is presented as practical and general.

Catalog record

CREATORSoheil Zibakhsh Shabgahi, Pedram Aghazadeh, Azalia Mirhosseini, Farinaz Koushanfar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08972

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ForTIFAI: Fending Off Recursive Training Induced Failure for AI Models Soheil Zibakhsh Shabgahi, Pedram Aghazadeh, Azalia Mirhosseini, Farinaz Koushanfar Abstract The increasing reliance on generative AI models has accelerated the generation rate of synthetic data, with some projections suggesting that most available new data for training could be machine-generated by 2030 Gartner, Inc. 2022 . This shift to a mainly synthetic content presents a critical challenge: repeated training in synthetic data leads to a phenomenon known as model collapse, where model performance degrades over generations of training, eventually rendering the models ineffective. Although prior studies have explored the causes and detection of model collapse, existing mitigation strategies remain limited. In this paper, we identify model overconfidence in their self-generated data as a key driver of collapse. Building on this observation, we propose a confidence-aware loss function that downweights high-confidence predictions during training. We introduce a novel loss function we call Truncated Cross Entropy (TCE). We demonstrate that TCE significantly delays model collapse in recursive training. We provide a model-agnostic framework that links the loss function design to model collapse mitigation and validate our approach both theoretically and empirically, showing that it can extend the model’s fidelity interval before collapse by more than 2.3 × 2.3\times . Finally, we show that our method generalizes across modalities. These findings suggest that the design of loss functions provides a simple yet powerful tool for preserving the quality of generative models in the era of increasing synthetic data. \svgpath svg_figures/ † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Generative models have become the foundation for modern AI applications in several modalities, including text, image, code, and audio. Large Language Models (LLMs) such as ChatGPT OpenAI et al. 2024 , LLaMA Grattafiori et al. 2024 , Phi Abdin et al. 2024 , and DeepSeek DeepSeek-AI et al. 2025 , as well as image generators DALL-E Ramesh et al. 2021 and Imagen Saharia et al. 2022 , all rely on large datasets scraped from the Web. As these models are continuously updated to reflect recent knowledge and linguistic patterns, the need for ever larger and frequently refreshed training corpora has grown substantially. However, this demand is colliding with a shift in the data landscape: synthetic content is increasingly populating the Internet, contaminating the very datasets used for model training. This shift raises fundamental concerns. Generative models trained on outputs from earlier generations can degrade over time, a phenomenon known as model collapse Shumailov et al. ; Shumailov et al. 2024 ; Dohmatob et al. 2024a . Even moderate levels of synthetic data (as little as 1%) have been shown to cause collapse, and scaling the model or the size of the dataset fails to reliably prevent it Dohmatob et al. 2024b . Importantly, this is not a hypothetical scenario: recent analysis of the LAION-5B dataset Alemohammad et al. 2023 , a cornerstone for many generative models, revealed measurable contamination from synthetic sources. 1 1 1 Following prior works Shumailov et al. ; Shumailov et al. 2024 , we use the term contamination or poisoning to describe the recursive use of synthetic data in training, which leads to degradation in performance over generations. As synthetic data becomes dominant, safeguarding generative model stability becomes an urgent challenge. Several approaches have been explored to mitigate model collapse. Some studies combine synthetic and real data to slow degradation Gerstgrasser et al. 2024 ; others rely on post hoc supervision, such as SIMS Alemohammad et al. 2024 , using labeled synthetic data as negative signal to improve generations or explore decoding strategies Drayson et al. 2025 . In practice, however, distinguishing real from synthetic data remains a major challenge. Watermarking methods Zhang and Koushanfar 2024 ; Kirchenbauer et al. 2024 have been proposed to help identify model-generated content, but they are not widely adopted. More fundamentally, recent critiques Schaeffer et al. 2025 highlight that many prior studies simulate collapse in unrealistic settings, such as using purely synthetic training data, limiting the applicability of their conclusions. Despite growing interest in this issue, the role of the loss function, central to how models learn, has received little attention in the context of model collapse. In this work, we present ForTIFAI, showing that model collapse can be mitigated through principled modifications to training loss. We argue that generative models tend to be overconfident in their synthetic predictions (as depicted in Figure 1 ), causing feedback loops that degrade performance. This issue is further amplified by approximation errors and finite sampling effects during training, which cause models to concentrate probability mass on a shrinking subset of the true data distribution over generations Shumailov et al. 2024 ; Dohmatob et al. 2024a . To counter this, we introduce Truncated Cross Entropy (TCE), a novel confidence-aware loss function that reduces the weight of high-confidence predictions during training by masking predictions above a confidence threshold. TCE is model-agnostic and easy to implement on top of Cross Entropy (CE), making it attractive for real-world deployment. In brief, the contributions of ForTIFAI are as follows: • We introduce a loss-based, model-agnostic method for mitigating model collapse, explicitly addressing sampling bias identified in prior studies. • We propose Truncated Cross Entropy (TCE), a novel confidence-aware loss function that downweights high-confidence tokens during training, and demonstrate that it extends model fidelity by more than 2.3\times . • We design a comprehensive evaluation framework grounded in practical and evolving data regimes, including datasets where new real-world content continues to appear, some content is regenerated multiple times while other parts remain untouched, and model quality is measured by retained knowledge rather than token similarity. • We demonstrate the generality of our approach by applying it to diverse generative models, including Gaussian Mixture Models (GMMs) and Variational Autoencoders (VAEs), confirming its effectiveness beyond LLMs. 2 Method In this section, we first describe the Model Collapse phenomena. We then provide intuition about the correlation between model collapse and token prediction confidence 2 2 2 Throughout this paper, we define a model’s confidence in tokens as the predicted probability assigned to them. in Large Language Models (LLMs). Building upon this intuition, we derive a class of confidence-aware loss functions to thwart autophagy Alemohammad et al. 2023 in generative models. 2.1 Problem Definition Model Collapse is the gradual degradation that occurs when a model’s own outputs pollute the training data of its successors. Each new model then learns from these possibly flawed examples, causing a drift from the true data distribution Shumailov et al. . During the generation phase, statistical approximations, limited functional expressiveness, and approximation errors introduce non-zero probabilities of distributional shifts in the model’s outputs. (a) Confidences over Wikitext (b) Confidences over re-generated Wikitext Figure 1 : Models display higher confidence in their own generated data when recursively trained. These charts illustrate the probabilities assigned by the LLaMA-3.2-1B model and a version fine-tuned on a fraction of Wikitext. The synthetic dataset in this case is generated by the fine-tuned model. The fine-tuned model shows higher confidence on the portion of the Wikitext it was trained on as well as the entire dataset generated by it. In contrast, the original model demonstrates similar

Read the complete text