MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation
Yiren Song, Cheng Liu, Mike Zheng Shou · 2025
Read MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation by Yiren Song, Cheng Liu, Mike Zheng Shou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01572.txt.
Introduction
Yiren Song, Cheng Liu, and Mike Zheng Shou build MakeAnything, a tool that writes step-by-step guides as image sequences. Their twenty twenty-five paper takes on a hard creative task. Guides need steps that stay in order and look like one set.
The team first built a data set of more than twenty-four thousand sequences across twenty-one tasks. Their model rests on a diffusion transformer, tuned to use context from past steps. A lopsided low-rank tuning keeps broad skill while fitting the guide task. Results span crafts, cooking, and other fields. Readers see how one system can draft clear how-to sequences in many domains.
Catalog record
| CREATOR | Yiren Song, Cheng Liu, Mike Zheng Shou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-01572
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation Yiren Song, Cheng Liu, Mike Zheng Shou Abstract A hallmark of human intelligence is the ability to create complex artifacts through structured multi-step processes. Generating procedural tutorials with AI is a longstanding but challenging goal, facing three key obstacles: (1) scarcity of multi-task procedural datasets, (2) maintaining logical continuity and visual consistency between steps, and (3) generalizing across multiple domains. To address these challenges, we propose a multi-domain dataset covering 21 tasks with over 24,000 procedural sequences. Building upon this foundation, we introduce MakeAnything, a framework based on the diffusion transformer (DIT), which leverages fine-tuning to activate the in-context capabilities of DIT for generating consistent procedural sequences. We introduce asymmetric low-rank adaptation (LoRA) for image generation, which balances generalization capabilities and task-specific performance by freezing encoder parameters while adaptively tuning decoder layers. Additionally, our ReCraft model enables image-to-process generation through spatiotemporal consistency constraints, allowing static images to be decomposed into plausible creation sequences. Extensive experiments demonstrate that MakeAnything surpasses existing methods, setting new performance benchmarks for procedural generation tasks. Code is released at https://github.com/showlab/MakeAnything Figure 1 : We introduce MakeAnything, a tool that realistically and logically generates step-by-step procedural tutorial for activities such as painting, crafting, and cooking, based on text descriptions or conditioned images. 1 Introduction A defining characteristic of human intelligence—and a key differentiator from other species—is the capacity to create complex artifacts through structured step-by-step processes. In computer vision, generating such procedural sequences for tasks like painting, crafting, product design, and culinary arts remains a significant challenge. The core difficulty lies in producing multi-step sequences that maintain logical continuity and visual consistency, requiring models to both capture intricate visual features and understand causal relationships between steps. This challenge becomes particularly pronounced when handling diverse domains and styles without compromising generation quality—a problem space that remains underexplored. Existing research primarily focuses on decomposing painting processes, with early methods employing reinforcement learning/optimization algorithms through stroke-based rendering to approximate target images. Subsequent works like ProcessPainter ( Song et al., 2024a ) and PaintsUndo ( Team, 2024 ) utilize temporal models on synthetic datasets, while Inverse Painting ( Chen et al., 2024 ) redicts the order of human painting, generating the painting process by region. However, these approaches remain limited to single-task scenarios and exhibit poor cross-domain generalization. Furthermore, ProcessPainter’s Animatediff-based framework constrains modifications to minor motion adjustments, making it unsuitable for categories requiring structural transformations (e.g., recipes or crafts). Although Diffusion Transformer (DIT) ( Peebles & Xie, 2023 ) -based video generation models can produce long sequences, their effectiveness is hindered by distribution shifts in training data when generating complex procedural workflows. We posit that replicating human creative intelligence requires both high-quality multi-task procedural data and advanced methodology design.. To this end, we curate a comprehensive multi-domain dataset spanning 21 categories (including painting, crafts, SVG design, LEGO assembly, and cooking) with over 24,000 procedurally annotated sequences—the largest such collection for step-by-step creation tasks. Methodologically, we propose MakeAnything, a novel framework that harnesses the in-context capabilities of Diffusion Transformers (DIT) through LoRA fine-tuning to generate high-quality instructional sequences. Addressing the challenge of severe data scarcity (some categories have as few as 50 data entries.) and imbalanced distributions, we employ an asymmetric low-rank adaptation (LoRA) ( Zhu et al., 2024 ; Hu et al., 2022 ) strategy for image generation. This approach combines a pretrained encoder on large-scale data with a task-specific fine-tuned decoder, achieving an optimal balance between generalization and domain-specific performance. To address practical needs for reverse-engineering creation processes, we develop the ReCraft Model—an efficient controllable generation method that decomposes static images into step-by-step procedural sequences. Building upon the pretrained Flux model with minimal architectural modifications, ReCraft introduces an image-conditioning mechanism where clean latent tokens from the target image (encoded via VAE) guide the denoising of noisy intermediate frames through multi-modal attention. Remarkably, this lightweight adaptation enables efficient training with limited data—ReCraft achieves robust performance with just hundreds or even dozens of process sequences per task. During inference, the model recursively predicts preceding frames over concatenated latent representations, effectively reconstructing the creation history from static artworks. In summary, our contributions are as follows: 1. Unified Procedural Generation Framework : We introduce MakeAnything, the first DIT-based architecture enabling cross-domain procedural sequence synthesis, supporting both text-to-process and image-to-process generation paradigms. 2. Technical Innovations : We employ an asymmetric LoRA architecture for cross-domain generalization and the ReCraft Model for image-conditioned process reconstruction with limited training data. 3. Dataset Contribution : We propose a multi-domain procedural dataset (21 categories, 24K+ sequences) with hierarchical annotations, significantly advancing research in procedural understanding and generation. 2 Related Work 2.1 Diffusion Models Diffusion probability models ( Song et al., 2020 ; Ho et al., 2020 ) are advanced generative models that restore original data from pure Gaussian noise by learning the distribution of noisy data at various levels of noise. Their powerful capability to adapt to complex data distributions has led diffusion models to achieve remarkable success across several domains including image synthesis ( Rombach et al., 2022 ; Peebles & Xie, 2023 ) , image editing ( Brooks et al., 2023 ; Hertz et al., 2022 ; Zhang et al., 2024c ; Zhang et al., 2024d ) , and video gneration ( Guo et al., 2023 ; Blattmann et al., 2023 ; Song et al., 2024a ) , evaluation ( Song et al., 2024b ) . Stable Diffusion ( Rombach et al., 2022 ) (SD), a notable example, utilizes a U-Net architecture and extensively trains on large-scale text-image datasets to iteratively generate images with impressive text-to-image capabilities. The Diffusion Transformer (DiT) model ( Peebles & Xie, 2023 ) , employed in architectures like FLUX.1 ( AI, 2024 ) , Stable Diffusion 3 ( Esser et al., 2024 ) , and PixArt (pixart), uses a transformer as the denoising network to iteratively refine noisy image tokens. Customized generation methods enable flexible customization of concepts and styles by fine-tuning U-Net ( Ruiz et al., 2023 ) or certain parameters ( Hu et al., 2022 ; Kumari et al., 2023 ) , alongside trainable tokens. Training-free customization methods ( Ye et al., 2023 ; Zhang et al., 2024a ; Zhang et al., 2024b ) leverage pre-trained CLIP ( Radford et al., 2021 ) encoders to extract image features for efficient customized generation. Figure 2 : The MakeAnything framework comprises two core components: (1) an Asymmetric LoRA module that generates diverse creation processes from text prompts through asymmetric LoRA, and (2) the ReCraft Model, which constructs an image-conditioned base model by merging pretrained LoRA weights with the Flux foundation model, enabling process prediction via injected visual tokens. 2.2 Controllable Generation in Diffusion Models Controllable generation has been extensively studied in the context of diffusion models. Text-to-image models ( Ho et al., 2020 ; Song et al., 2020 ) have established a foundation for conditional generation, while various approaches have been developed to incorporate additional control signals such as images. Notable methods include ControlNet ( Zhang & Agrawala, 2023 ) , enabling spatially aligned control in diffusion models, and T2I-Adapter ( Mou et al., 2023 ) , which improves efficiency with lightweight adapters. UniControl ( Zhao et al., 2023 ) uses Mixture-of-Experts (MoE)