Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising

Gongfan Fang, Xinyin Ma, Xinchao Wang · 2024

Read Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising by Gongfan Fang, Xinyin Ma, Xinchao Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05628.txt.

Description

Gongfan Fang, Xinyin Ma, and Xinchao Wang split one big image-generating transformer into a team of mixed experts. Diffusion models clean noise step by step, and the best results demand huge models with heavy training and running costs. Remix-DiT instead keeps a few basis models and blends them with learned weights to form a specialist for each noise stage. The blended model has the same shape as an ordinary one, so running it costs no extra.

The learned blend also spends capacity where each stage needs it most, which lifts picture quality. On ImageNet tests, the method matched or beat standard diffusion transformers and other multi-expert designs. The authors share their code openly. They frame mixing as a cheaper road to expert-level generation than training every expert from scratch.

Catalog record

CREATORGongfan Fang, Xinyin Ma, Xinchao Wang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05628

Browse the full library · About the Tower of Babel