LegoDiffusion : Micro-Serving Text-to-Image Diffusion Workflows
Lingyun Yang †∗ , Suyi Li †∗# , Tianyu Feng † , Xiaoxiao Jiang † , Zhipeng Di, Weiyi Lu, Kan Liu, Yinghao Yu, Tao Lan, Guodong Yang, Lin Qu, Liping Zhang, Wei Wang † † Hong Kong University of Science and Technology Alibaba Group · 2026
Read LegoDiffusion : Micro-Serving Text-to-Image Diffusion Workflows by Lingyun Yang †∗ , Suyi Li †∗# , Tianyu Feng † , Xiaoxiao Jiang † , Zhipeng Di, Weiyi Lu, Kan Liu, Yinghao Yu, Tao Lan, Guodong Yang, Lin Qu, Liping Zhang, Wei Wang † † Hong Kong University of Science and Technology Alibaba Group (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08123.txt.
Introduction
Yang and 12 colleagues present LegoDiffusion, a micro-serving system for text-to-image workflows. Text-to-image generation runs a workflow of models centered on a base model. Current serving systems treat each workflow as a closed monolith, provisioning, placing, and scaling all part models, hiding internal flow, preventing model sharing, and forcing coarse-grained resource handling. LegoDiffusion splits a workflow into loosely coupled model nodes managed and scheduled apart. Open per-model inference handling unlocks cluster gains: per-model scaling, model sharing, and tuned model parallel scaling.
Catalog record
| CREATOR | Lingyun Yang †∗ , Suyi Li †∗# , Tianyu Feng † , Xiaoxiao Jiang † , Zhipeng Di, Weiyi Lu, Kan Liu, Yinghao Yu, Tao Lan, Guodong Yang, Lin Qu, Liping Zhang, Wei Wang † † Hong Kong University of Science and Technology Alibaba Group |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66693 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08123
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LegoDiffusion : Micro-Serving Text-to-Image Diffusion Workflows Lingyun Yang †∗ , Suyi Li †∗# , Tianyu Feng † , Xiaoxiao Jiang † , Zhipeng Di, Weiyi Lu, Kan Liu, Yinghao Yu, Tao Lan, Guodong Yang, Lin Qu, Liping Zhang, Wei Wang † † Hong Kong University of Science and Technology Alibaba Group Abstract . Text-to-image generation executes a diffusion workflow comprising multiple models centered on a base diffusion model. Existing serving systems treat each workflow as an opaque monolith, provisioning, placing, and scaling all constituent models together, which obscures internal dataflow, prevents model sharing, and enforces coarse-grained resource management. In this paper, we make a case for micro-serving diffusion workflows with LegoDiffusion , a system that decomposes a workflow into loosely coupled model-execution nodes that can be independently managed and scheduled. By explicitly managing individual model inference, LegoDiffusion unlocks cluster-scale optimizations, including per-model scaling, model sharing, and adaptive model parallelism. Collectively, LegoDiffusion outperforms existing diffusion workflow serving systems, sustaining up to 3 × \times higher request rates and tolerating up to 8 × \times higher burst traffic. * * footnotetext: Equal contribution; # Corresponding author 1. Introduction Text-to-image (T2I) generation using diffusion models enables the creation of high-quality, contextually accurate images from textual description ( 73 ; 32 ; 89 ; 51 ; 52 ; 45 ) . A typical T2I generation workflow integrates a base diffusion model with multiple adapter models to form a pipeline. Execution begins with text encoders that convert prompts into embeddings (Fig. 1 -top). Conditioned on the embeddings, the diffusion model iteratively generates latent representations via a denoising process, which are subsequently decoded as the final image. To refine visual attributes such as composition or artistic styles, T2I workflows increasingly incorporate adapter models, such as ControlNet ( 89 ) and LoRA ( 26 ) , alongside the base model ( 37 ; 40 ) . By augmenting the diffusion process (Fig. 1 ), these adapters enable fine-grained alignment with user intent and aesthetics ( 89 ; 26 ; 78 ; 90 ) . Figure 1. Top : a basic diffusion workflow using Flux-Dev ( 33 ) . Middle : add ControlNets ( 89 ) alongside the base diffusion model, which process an additional input reference image and pass the intermediaries to diffusion model to control the composition in image generation. Bottom : Further add LoRA ( 26 ) to change image styles by patching LoRA weights onto diffusion model weights. Despite the modular nature of diffusion workflows, existing inference systems, such as HuggingFace Diffusers ( 14 ) and ComfyUI ( 6 ) , primarily employ a monolithic serving practice, where the entire workflow, comprising the base model and all associated adapters, is encapsulated into a monolithic instance for provisioning and scheduling. The system treats these workflow instances as opaque black boxes running on a fixed set of GPUs, while remaining oblivious to the workflow’s internal model executions and data exchanges. While operationally simple, monolithic serving introduces fundamental inefficiencies. First, it forces coarse-grained scaling : if a single model becomes a bottleneck, the system must replicate the entire workflow. Second, isolated monoliths preclude model sharing . Production traces ( 37 ; 40 ) show that popular diffusion backbones and adapters are frequently reused across workflows. However, monolithic serving forces each workflow instance to maintain its own model copies, leading to redundant memory footprints. Third, treating workflows as opaque black boxes hides internal data dependencies, preventing the system from automatically optimizing resource allocation and pipelining. Finally, tight coupling increases fragility: a single model failure crashes the entire workflow. To address these limitations, we argue that the schedulable unit for inference should be the individual model execution node, not the entire diffusion workflow. Instead of monolithic instances, the system should decompose workflows into loosely-coupled microservices—each encapsulating a specific component like a text encoder, diffusion backbone, or adapter. This micro-serving architecture directly addresses the inefficiencies of monolithic serving. First, it enables fine-grained scaling : individual models can scale elastically based on real-time demand, eliminating the resource waste of replicating the entire pipeline. Second, it facilitates cross-workflow model sharing : distinct workflows can multiplex shared models, such as a common base model, avoiding redundant memory footprints. Third, by making model execution and data flow explicit, the system regains visibility into the computation graph, enabling automated optimization of resource allocation and pipelining. Decoupling the workflow also enables fault isolation and fast failure recovery. However, realizing micro-serving for diffusion workflows presents non-trivial systems challenges. While prior frameworks have successfully applied micro-serving to CPU-centric data analytics ( 82 ; 59 ; 87 ; 69 ; 80 ) and LLM-based agentic workflows ( 46 ; 62 ; 34 ; 42 ) , applying this paradigm to diffusion pipelines requires overcoming hurdles that these systems cannot handle. First, diffusion workflows exhibit complex, iterative data dependencies between base models and adapters, which cannot be expressively defined or easily supported in existing frameworks. Second, decoupling these tightly integrated model workflows necessitates massive, latency-sensitive tensor communications across GPUs. Existing LLM or CPU micro-serving frameworks lack an efficient data plane to manage these high-bandwidth transfers. In this paper, we present LegoDiffusion , a system purpose-built for the efficient micro-serving of diffusion workflow. LegoDiffusion introduces three key designs: Programming Interface & Compilation. LegoDiffusion provides a Python-embedded domain-specific language (DSL) for composing diffusion workflows. The DSL exposes primitives for model initialization, inference, and diffusion-specific operations for LoRA and ControlNet application ( 26 ; 89 ) . To provide a unified support for diverse community-developed models ( 29 ) , LegoDiffusion wraps each model behind a standardized interface that encapsulates its native loading and inference logic. All primitives enforce strict input/output typing, making data dependencies explicit and catching errors at compile time. A graph compiler translates the workflow composition into a directed acyclic graph (DAG) of loosely coupled workflow nodes . Each node represents a discrete model inference operator that can be independently provisioned and scheduled–the fundamental unit of micro-serving. Runtime & Data Plane. At runtime, LegoDiffusion applies lazy execution : upon each request, it analyzes the workflow DAG and dynamically recomposes the compute graph—inserting or substituting nodes—to apply diffusion-specific optimizations. Decomposing workflows into distributed nodes, however, introduces high-bandwidth tensor transfers with complex synchronization requirements (e.g., forwarding ControlNet intermediates at specific denoising layers). To handle this, we design a distributed data engine atop NVSHMEM ( 48 ) that enables GPU-direct, zero-copy tensor movement over high-speed interconnects. The engine provides two fetch modes— eager and deferred —so that tensors arrive precisely when needed without stalling execution. These mechanisms are transparent to developers: once the compiler produces the workflow DAG, the data engine automatically orchestrates all inter-node tensor transfers. Workflow Node Scheduling. The LegoDiffusion scheduler maps workflow nodes onto distributed executors using three strategies that exploit micro-serving’s decomposition. First, it enforces model-granular scaling : rather than replicating entire workflows, LegoDiffusion scales only the bottleneck models, avoiding redundant resource provisioning. Second, because a loaded model is workflow-agnostic, the scheduler preferentially dispatches nodes to executors that already hold the required model state, enabling multi-tenant model sharing . Third, LegoDiffusion employs adaptive parallelism : it dynamically adjusts model parallelism at scheduling time based on real-time cluster availability, right-sizing resource allocation to maximize throughput without incurring queuing delays. We