Decentralized Diffusion Models
David McAllister, Matthew Tancik, Jiaming Song, Angjoo Kanazawa · 2025
Read Decentralized Diffusion Models by David McAllister, Matthew Tancik, Jiaming Song, Angjoo Kanazawa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05450.txt.
Introduction
McAllister, Tancik, Song, and Kanazawa propose training image-generating AI in separate pieces in this two thousand twenty-five paper. Standard training spreads work over thousands of linked chips that must constantly synchronize, which only huge centralized clusters can support.
Instead, their decentralized diffusion models train a set of expert models on separate slices of the data, each cluster working fully on its own. A light router blends the experts when generating, and the team shows the blend optimizes the same goal as one model trained on everything. On standard image sets the approach beat ordinary training for the same compute, and they scaled it to twenty-four billion parameters on just eight machines in under a week. The method opens big-model training to smaller, cheaper setups.
Catalog record
| CREATOR | David McAllister, Matthew Tancik, Jiaming Song, Angjoo Kanazawa |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05450
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decentralized Diffusion Models David McAllister, Matthew Tancik, Jiaming Song, Angjoo Kanazawa Abstract Large-scale AI model training divides work across thousands of GPUs, then synchronizes gradients across them at each step. This incurs a significant network burden that only centralized, monolithic clusters can support, driving up infrastructure costs and straining power systems. We propose Decentralized Diffusion Models, a scalable framework for distributing diffusion model training across independent clusters or datacenters by eliminating the dependence on a centralized, high-bandwidth networking fabric. Our method trains a set of expert diffusion models over partitions of the dataset, each in full isolation from one another. At inference time, the experts ensemble through a lightweight router. We show that the ensemble collectively optimizes the same objective as a single model trained over the whole dataset. This means we can divide the training burden among a number of “compute islands,” lowering infrastructure costs and improving resilience to localized GPU failures. Decentralized diffusion models empower researchers to take advantage of smaller, more cost-effective and more readily available compute like on-demand GPU nodes rather than central integrated systems. We conduct extensive experiments on ImageNet and LAION Aesthetics, showing that decentralized diffusion models FLOP-for-FLOP outperform standard diffusion models. We finally scale our approach to 24 billion parameters, demonstrating that high-quality diffusion models can now be trained with just eight individual GPU nodes in less than a week. Figure 1 : Decentralized Diffusion Models (DDM). Left: Existing diffusion models (monolithic) require synchronized, centralized training across thousands of GPUs, making high-quality training systems expensive and inaccessible. Right: DDM divides a diffusion model into an ensemble of expert models, each trained on its own data cluster in complete isolation. This ensemble collectively optimizes the same diffusion objective as a single model trained on all the data. This enables flexible training across diverse cloud or academic compute facilities. At inference, the ensemble delivers improved performance at the same FLOP-cost, making high-quality diffusion model training more efficient and accessible. See Figure 2 for large-scale DDM model samples. Figure 2 : Decentralized diffusion models train on readily-available hardware and generate high quality, diverse images. We present selected samples from our 8x3B parameter model. 1 Introduction Diffusion models achieve breakthrough results in image generation [ 39 , 40 ] , video modeling [ 4 ] , and robotic control through diffusion policies [ 10 ] . However, these models continue to demand greater and greater training compute. Even early successes in image diffusion underscored the immense computational requirements, with Stable Diffusion 1.5’s training consuming over 6,000 A100 GPU days [ 34 , 7 ] . Video diffusion models demand a step-function increase in these resources. For instance, Meta’s Movie Gen trains on up to 6,114 H100 GPUs [ 35 ] —nearly GPT-3 scale [ 6 ] . As these models continue to grow in size and capability, they encounter significant systems-level challenges that mirror those faced in LLM training. High-bandwidth interconnect becomes a critical bottleneck, storage systems must handle massive streaming datasets and persistent hardware failures can crash or silently slow the training process [ 1 , 14 ] . The result is a fragile integrated system where performance and reliability depend on complex interactions between arrays of accelerators and networking hardware. This is challenging and costly for large industry labs—untenable for academics. We present Decentralized Diffusion Models, a scalable method for distributing the modeling burdening of diffusion across independent expert models, each trained on its own “compute island” with no cross-communication. This enables training with scattered resources, leveraging cost-effective cloud compute or combining resources across multiple clusters. Decentralization is also helpful for training “foundation”-scale models [ 3 ] , where it is increasingly difficult to build datacenters with enough power capacity for the scale of modern training runs [ 31 ] . Independent training provides additional benefits, such as the ability to execute across heterogeneous hardware, making it possible to reuse existing training clusters alongside newer infrastructure. Decentralized diffusion models employ a new training objective, Decentralized Flow Matching (DFM), which partitions the training data into K groups and trains a dedicated diffusion model over each. We refer to these specialized models as experts, inspired by the Mixture of Experts (MoE) literature [ 16 ] . An independently trained router model orchestrates these experts, determining which are most appropriate at test-time. We show that this ensemble of expert models collectively optimizes the same global objective as a single model trained on the entire dataset. During each inference step, the router predicts each expert’s relevance to the input noise and condition. The expert predictions are then linearly combined, weighed by their associated relevance scores. Experts learn to specialize, so many are irrelevant to a given input, and it is more efficient to use a subset of them. If not all experts are selected, it serves as a sparse model, leveraging selective computation similar to MoE. Sparse inference is compute-efficient but remains memory-intensive. We demonstrate that we can distill experts into a single dense model, achieving convenient inference while preserving sample quality. We evaluate our approach on ImageNet and a filtered subset of LAION-Aesthetics, where we compare DDMs against existing monolithic diffusion model training. We systematically vary the number of experts to analyze its impact on downstream generation tasks, where we find that DDMs with eight experts consistently achieve the best performance, even outperforming a single model trained with the same compute. This is made possible by the additional parametrization provided by multiple specialized experts to outperform a generalist model. Finally, we demonstrate the scaling ability of DDMs by training a large decentralized model with eight 3-billion parameter experts, resulting in high definition image generation, as shown in Figure 2 . Decentralized Diffusion Models also integrate easily into existing diffusion training environments. In practice, DDMs involve clustering a dataset then training a standard diffusion model on each cluster. This means nearly everything from existing diffusion infrastructure can be reused—training code, dataloading, systems optimizations, noise schedules and architectures. See our blog post for a visual walkthrough and pseudocode. 2 Related Works Accelerating Diffusion Models Flow matching [ 27 , 28 ] generalizes diffusion models [ 44 , 45 , 21 , 46 ] and makes tractable large-scale training of continuous normalizing flows (CNFs) [ 8 ] . Diffusion and flow matching models produce state of the art generations in high dimensional continuous spaces but are typically expensive to train to high quality. Recent works propose token masking [ 43 ] and aligning internal states with pretrained image representation models [ 52 ] to accelerate the model learning process. PixArt- \mathbf{\alpha} [ 7 ] shows that curated datasets with synthetic captions reduce training costs. We explore a complementary direction that decentralizes the training of the model across multiple GPU clusters without cross-communication, thereby increasing the robustness and efficiency of training. Existing approaches all use data parallel training, which distributes batches across GPUs and synchronizes gradients to produce larger effective batch sizes. Our work introduces an orthogonal form of parallelism that partitions the model’s training across isolated compute centers with no gradient synchronization. Our approach is complementary with existing techniques—in fact, we employ Fully Sharded Data Parallel (FSDP) [ 53 ] training within each cluster while maintaining isolation between them in our