Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising
Gongfan Fang, Xinyin Ma, Xinchao Wang · 2024
Read Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising by Gongfan Fang, Xinyin Ma, Xinchao Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05628.txt.
Description
Gongfan Fang, Xinyin Ma, and Xinchao Wang split one big image-generating transformer into a team of mixed experts. Diffusion models clean noise step by step, and the best results demand huge models with heavy training and running costs. Remix-DiT instead keeps a few basis models and blends them with learned weights to form a specialist for each noise stage. The blended model has the same shape as an ordinary one, so running it costs no extra.
The learned blend also spends capacity where each stage needs it most, which lifts picture quality. On ImageNet tests, the method matched or beat standard diffusion transformers and other multi-expert designs. The authors share their code openly. They frame mixing as a cheaper road to expert-level generation than training every expert from scratch.
Catalog record
| CREATOR | Gongfan Fang, Xinyin Ma, Xinchao Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05628