Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models 1051
Zizhao Hu, Mohammad Rostami, Jesse Thomason · 2025
Read Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models 1051 by Zizhao Hu, Mohammad Rostami, Jesse Thomason (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08803.txt.
Introduction
Zizhao Hu, Mohammad Rostami, and Jesse Thomason study model collapse when AI systems train on each other's output, in a 2025 paper. Collapse is the slow decay seen when models feed on synthetic data. Past work covered single text or image models only. Real agents mix vision and language and swap data. The team ran recursive generate-and-train loops across model types.
Collapse looked different in the mixed setting. Image-caption alignment sometimes improved while variety grew in odd ways. Other traits still degraded across loops. Several defenses worked against the decay. Larger decoding budgets helped stability. Mixing diverse models in the loop also helped. Freezing a relabeling model curbed drift further. The authors distill early guidelines for self-improving systems. Careful data curation remains the central defense.
Catalog record
| CREATOR | Zizhao Hu, Mohammad Rostami, Jesse Thomason |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08803
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models 1051 Zizhao Hu, Mohammad Rostami, Jesse Thomason Abstract . Recent research has highlighted the risk of generative model collapse, where performance progressively degrades when continually trained on self-generated data. However, existing exploration on model collapse is limited to single, unimodal models, limiting our understanding in more realistic scenarios, such as diverse multi-modal AI agents interacting autonomously through synthetic data and continually evolving. We expand the synthetic data training and model collapse study to multi-modal vision-language generative systems, such as vision-language models (VLMs) and text-to-image diffusion models, as well as recursive generate-train loops with multiple models. We find that model collapse, previously observed in single-modality generative models, exhibits distinct characteristics in the multi-modal context, such as improved vision-language alignment and increased variance in VLM image-captioning task. Additionally, we find that general approaches such as increased decoding budgets, greater model diversity, and relabeling with frozen models can effectively mitigate model collapse. Our findings provide initial insights and practical guidelines for reducing the risk of model collapse in self-improving multi-agent AI systems and curating robust multi-modal synthetic datasets. Figure 1. Recursive finetuning configurations for investigating model collapse: (1) Single model recursive finetuning (“T2I” stands for text-to-image diffusion model), (2) Finetuning with VLM or diffusion relabeling, and (3) Joint recursive finetuning. (1) allows independent study of the effect of recursive training on VLMs and text-to-image diffusion models. (2) enables information integration from another generative paradigm, which simulates a multi-agent environment where a frozen grounding model is available and used to label synthetic data. (3) simulates a multi-agent environment where all models have full freedom to update their weights using available synthetic data. Diagram showing three different experimental setups for recursive finetuning of vision-language models. 1. Introduction The key to training large-scale generative AI models is the availability of large-scale training datasets. As models grow in size and capability, the amount of available human-authored data for training becomes a significant bottleneck. This bottleneck has prompted the exploration of synthetic data for training AI models. In the realm of LLMs, synthetic data has been used for pretraining ( Gunasekar et al. 2023 ; Li et al. 2023a ; Abdin et al. 2024 ) , continued pre-training ( Yang et al. 2024 ) , supervised finetuning for general tasks ( Li et al. 2024 ) , or specialized tasks such as math ( Yu et al. 2024 ; Luo et al. 2023a ; Gao et al. 2023 ; Trinh and Le 2024 ) and coding ( Wang et al. 2024 ; Luo et al. 2023b ) , and post-training RL ( DeepSeek-AI et al. 2025 ; Grattafiori et al. 2024 ; Bai et al. 2023a ; Lambert et al. 2025 ) . Other than LLMs, synthetic data has also been used to train VLMs ( Liu et al. 2023 ) . In continual learning, synthetic data has also been used to reduce catastrophic forgetting ( Shin et al. 2017 ) . In model training, synthetic data offers advantages in terms of quantity, controllability, and privacy. However, because generative models are often used to generate synthetic data for training generative models, model collapse through recursive training on self-generated data becomes a risk ( Shumailov et al. 2023 ; Alemohammad et al. 2023 ) . We directly address several limitations of the existing literature studying such generative model collapse. In particular, existing works: (1) focus on characterizing model collapse in single-modality tasks (e.g., language generation or unconditional image generation), providing limited insight into how collapse manifests in multi-modal generative systems; (2) focus on a single model’s self-consuming training loop, without providing insights on more realistic scenarios where multiple models, from multiple generative paradigms, are within such a loop; and (3) offer solutions to model collapse primarily dependent on mixing synthetic data with human-authored data, without exploring how to improve the robustness of synthetic data. These limitations are particularly concerning given that a growing proportion of labeled web content is generated by two classes of multi-modal models: text-to-image diffusion models ( Rombach et al. 2022 ; Ramesh et al. 2021 ; Ramesh et al. 2022 ; Betker et al. [n. d.] ) and VLMs ( Liu et al. 2023 ; Li et al. 2023b ; OpenAI et al. 2024 ; Lin et al. 2024 ; Bai et al. 2023b ; Chen et al. 2025 ) . In this study, we address these limitations by expanding the investigation of synthetic data training and model collapse to multi-modal, multi-agent scenarios, answering: (1) How does recursive synthetic data training impact multi-modal generative systems? (2) What properties characterize multi-modal model collapse? (3) Do model interactions make collapse better or worse? (4) How can we improve generative synthetic data to mitigate model collapse? Through extensive experiments on VLMs and diffusion models, we make several key discoveries: (1) Balancing synthetic data bias-variance trade-off should be the focus for reducing model collapse rather than increasing variance only, since variance does not always collapse for certain tasks, such as VLM image captioning; (2) increased decoding budgets generate more robust synthetic datasets, which can mitigate model collapse despite no significant change in data quality; (3) commonly used statistical quality measures correlate with the robustness of synthetic data against model collapse; and (4) adding more models to the recursive training loop can make the model collapse worse, unless the added models are frozen and pretrained on human-authored data. Our analysis centers around recursive synthetic data training under two paradigms: image-captioning VLMs and text-to-image diffusion models. Our work provides insight into the generate-train self-consuming loop in a multi-modal and multi-agent environment. 2. Related work In this section, we briefly cover some representative existing works studying model training with synthetic data and model collapse, and provide background on multi-modal generative models. Model training using generative synthetic data. Generative models have been used for synthetic data curation for various downstream NLP tasks ( Yu et al. 2024 ; Luo et al. 2023a ; Gao et al. 2023 ; Trinh and Le 2024 ; Wang et al. 2024 ; Luo et al. 2023b ) and vision tasks ( Shin et al. 2018 ; Coyner et al. 2022 ; Daum et al. 2024 ) . Model collapse. Model collapse is a concept first coined by Shumailov et al. ( Shumailov et al. 2023 ) . It is a phenomenon in which models recursively trained on self-generated data experience performance degradation. It is hypothesized that model collapse happens because synthetic data lacks the diversity and nuances of real-world data. This issue has been identified in both language models ( Shumailov et al. 2024 ; Shumailov et al. 2023 ; Seddik et al. 2024 ) and image-generation models ( Alemohammad et al. 2023 ) . These works characterize model collapse by reduced perplexity and increased repetition in outputs generated by LLMs and diversity and quality degradation in output images of diffusion models. Addressing model collapse is critical, considering that using AI-generated synthetic data for training large AI models is becoming unavoidable as we exhaust human-authored data. Text-to-image diffusion models and VLMs. Text-to-image models like DALL-E ( Ramesh et al. 2021 ; Ramesh et al. 2022 ; Betker et al. [n. d.] ) , Stable