Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion
Zhongjie Duan, Hong Zhang, Yingda ChenModelScope Team, Alibaba Group · 2026
Read Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion by Zhongjie Duan, Hong Zhang, Yingda ChenModelScope Team, Alibaba Group (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24351.txt.
Introduction
Duan, Zhang, and Chen present Diffusion Templates, a plugin framework splitting inference from capability injection. Solo systems with clashing lines and hooks make reuse, transfer, and mixing hard. The framework has three parts: Template models mapping inputs to capability forms, a Template cache, and a Template pipeline. The pipeline loads, merges, and injects caches. The systems-level link lets carriers such as KV-Cache and LoRA work under one idea. A model zoo spans shape control, brightness, color, editing, super-detail, sharpness, looks, guide, fill-in, and age control.
Catalog record
| CREATOR | Zhongjie Duan, Hong Zhang, Yingda ChenModelScope Team, Alibaba Group |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35045 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.24351
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion Zhongjie Duan, Hong Zhang, Yingda ChenModelScope Team, Alibaba Group Abstract Controllable diffusion methods have substantially expanded the practical utility of diffusion models, but they are typically developed as isolated, backbone-specific systems with incompatible training pipelines, parameter formats, and runtime hooks. This fragmentation makes it difficult to reuse infrastructure across tasks, transfer capabilities across backbones, or compose multiple controls within a single generation pipeline. We present Diffusion Templates, a unified and open plugin framework that decouples base-model inference from controllable capability injection. The framework is organized around three components: Template models that map arbitrary task-specific inputs to an intermediate capability representation, a Template cache that functions as a standardized interface for capability injection, and a Template pipeline that loads, merges, and injects one or more Template caches into the base diffusion runtime. Because the interface is defined at the systems level rather than tied to a specific control architecture, heterogeneous capability carriers such as KV-Cache and LoRA can be supported under the same abstraction. Based on this design, we build a diverse model zoo spanning structural control, brightness adjustment, color adjustment, image editing, super-resolution, sharpness enhancement, aesthetic alignment, content reference, local inpainting, and age control. These case studies show that Diffusion Templates can unify a broad range of controllable generation tasks while preserving modularity, composability, and practical extensibility across rapidly evolving diffusion backbones. All resources will be open sourced, including code 1 1 1 https://github.com/modelscope/DiffSynth-Studio , models 2 2 2 https://modelscope.cn/collections/DiffSynth-Studio/KleinBase4B-Templates , and datasets 3 3 3 https://modelscope.cn/collections/DiffSynth-Studio/ImagePulseV2 . 1 Introduction Diffusion models have rapidly become a dominant foundation for visual generation, spanning high-quality image synthesis, image editing, and increasingly video generation [ 32 , 27 , 11 , 26 , 4 , 40 ] . As these backbones improve, practical applications demand richer forms of control over structure, appearance, identity, editing intent, style, and other task-specific factors. This demand has driven a broad family of controllable generation methods, including parameter-efficient adaptation methods such as LoRA [ 16 ] , personalization methods such as Textual Inversion and DreamBooth [ 12 , 35 ] , and conditional control modules such as ControlNet, T2I-Adapter, and IP-Adapter [ 49 , 24 , 47 ] . These methods are highly effective, but they are usually developed as isolated systems built around particular model architectures, condition types, and training recipes. This fragmentation creates an increasingly important systems bottleneck for controllable diffusion. In training, different control methods often require different model modifications, parameterizations, preprocessing code, and optimization objectives, which makes it difficult to reuse infrastructure across tasks or transfer a capability from one backbone to another. In deployment, each method exposes its own runtime hooks and parameter formats, so integrating a new control often means editing the pipeline itself rather than simply loading a reusable module. The difficulty becomes more severe when multiple controls must be enabled together: their conditioning pathways may compete for the same internal activations, require incompatible input formats, or depend on ad hoc fusion logic, making conflict resolution and joint scheduling largely a handcrafted engineering problem. As a result, today’s controllable diffusion ecosystem remains powerful but poorly modularized. In this paper, we argue that controllable generation capabilities should be treated as reusable plugins rather than backbone-specific modifications. We present Diffusion Templates , a unified and open plugin framework that decouples base-model inference from controllable capability injection. Our central claim is that many controllable diffusion methods can be reformulated as independent capability modules with a common systems interface, without restricting their model architecture or the format of their control conditions. Under this view, the base diffusion model remains responsible for generation quality, while each controllable capability is packaged as a Template model that can be trained, loaded, and composed independently. The framework is organized around three components. A Template model takes task-specific input, such as structural signals, scalar attributes, reference images, or other control conditions, and converts it into an intermediate capability representation called a Template cache. This cache is intentionally defined at the interface level rather than tied to a single internal form, so in practice it can be realized through different mediating representations, such as KV-Cache [ 21 ] , LoRA [ 16 ] , or other possible capability states. A Template pipeline then loads one or more Template models, collects their emitted caches, and injects them into the base diffusion runtime during generation. This separation cleanly factorizes controllable generation into two layers: capability construction and capability consumption. Compared with prior controllable diffusion models such as ControlNet or IP-Adapter, the key difference is that Diffusion Templates does not prescribe a specific control architecture or a fixed condition format. Instead, it defines a general interface through which heterogeneous control modules can interact with the same diffusion backbone while preserving strong empirical performance. Our design is loosely inspired by plugin abstractions in LLM systems, where standardized interfaces have made it possible to extend a strong base model with independently packaged capabilities [ 2 , 3 ] . We adopt this systems principle for diffusion models, but our focus is not on analogy for its own sake. Rather, the motivation is practical: once controllable capabilities are exposed through a stable interface, they become easier to train, reuse, combine, schedule, and maintain across a rapidly evolving family of diffusion backbones. Based on this framework, we train and release a diverse model zoo of Template models spanning structural control, brightness adjustment, color adjustment, image editing, super-resolution, sharpness enhancement, aesthetic alignment, content reference, local inpainting, and age control. Together, these case studies cover heterogeneous inputs, lightweight attribute controls, and more demanding image-conditioned tasks under the same runtime abstraction. They show that Diffusion Templates can unify a wide range of controllable generation capabilities without repeatedly redesigning the underlying diffusion pipeline. In summary, this paper makes the following contributions: • We propose Diffusion Templates, a unified plugin framework for controllable diffusion models that decouples base-model inference from capability injection and provides a common interface for training, loading, and composing heterogeneous control modules. • We train and release a diverse set of Template models spanning structural control, brightness adjustment, color adjustment, image editing, super-resolution, sharpness enhancement, aesthetic alignment, content reference, local inpainting, and age control, demonstrating the generality and practical potential of the framework across varied controllable generation tasks. 2 Related Work 2.1 Diffusion Foundation Models Diffusion foundation models have rapidly progressed from early denoising formulations (DDPM [ 14 ] , DDIM [ 37 ] ) to large-scale latent and transformer-based foundation models. A key milestone is LDM [ 32 ] , which established the practical latent diffusion paradigm and made high-quality generation computationally feasible at scale. Building on this line of work, the Stable Diffusion family has evolved from early Stable Diffusion releases to SD-XL [ 27 ] and Stable Diffusion 3 [ 11 ] , continuously improving semantic alignment, typography, and high-resolution synthesis quality. At the architectural level, DiT (Diffusion Transformer) [ 26 ] further accelerated the shift toward transformer-native diffusion backbones. In parallel, the open ecosystem has become increasingly diverse, with strong image-generation foundations such as FLUX [ 4 ] , Hunyuan-Image [ 23 ] , PixArt [ 7 ] , SANA [ 44 ] , and Qwen-Image [