Accelerating Controllable Generation via Hybrid-grained Cache

Lin Liu, Huixia Ben, Shuo Wang, Jinda Lu, Junxiang Qiu, Shengeng Tang, Yanbin Hao · 2025

Read Accelerating Controllable Generation via Hybrid-grained Cache by Lin Liu, Huixia Ben, Shuo Wang, Jinda Lu, Junxiang Qiu, Shengeng Tang, Yanbin Hao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11031.txt.

Introduction

Lin Liu, Huixia Ben, and their colleagues present a caching scheme that speeds up visual generation guided by extra control input, such as an edge map or pose. Control adds cost at every step, so guided models tend to run slowly. Reusing past work where nothing has changed is a natural way to cut that cost.

Their hybrid cache works at two scales. At the block scale it skips repeated work in whole encoder and decoder blocks, and at the prompt scale it reuses attention maps across nearby steps. On four test sets it kept visual quality close while cutting cost sharply, with one large saving on a segmentation benchmark.

Catalog record

CREATORLin Liu, Huixia Ben, Shuo Wang, Jinda Lu, Junxiang Qiu, Shengeng Tang, Yanbin Hao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11031

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerating Controllable Generation via Hybrid-grained Cache Lin Liu, Huixia Ben, Shuo Wang, Jinda Lu, Junxiang Qiu, Shengeng Tang, Yanbin Hao Abstract Controllable generative models have been widely used to improve the realism of synthetic visual content. However, such models must handle control conditions and content generation computational requirements, resulting in generally low generation efficiency. To address this issue, we propose a H ybrid- G rained C ache ( HGC ) approach that reduces computational overhead by adopting cache strategies with different granularities at different computational stages. Specifically, (1) we use a coarse-grained cache (block-level) based on feature reuse to dynamically bypass redundant computations in encoder-decoder blocks between each step of model reasoning. (2) We design a fine-grained cache (prompt-level) that acts within a module, where the fine-grained cache reuses cross-attention maps within consecutive reasoning steps and extends them to the corresponding module computations of adjacent steps. These caches of different granularities can be seamlessly integrated into each computational link of the controllable generation process. We verify the effectiveness of HGC on four benchmark datasets, especially its advantages in balancing generation efficiency and visual quality. For example, on the COCO-Stuff segmentation benchmark, our HGC significantly reduces the computational cost (MACs) by 63% (from 18.22T → \to 6.70 T↓), while keeping the loss of semantic fidelity (quantized performance degradation) within 1.5%. 1 Introduction Figure 1: We visualize the intermediate results of adding conditions at different steps in the controllable generative model: (a) adding throughout steps, (b) adding only in the first ten steps, and (c) adding only after ten steps. Controllable generative models ( 41 ; 11 ) incorporate conditional information into the content generation process, enabling more precise and tailored outputs compared to traditional generative methods. Thus, these models have gained widespread adoption ( 35 ; 34 ; 40 ) . However, during the denoising generation process, these models must simultaneously compute two critical components: the control module and the generative module. This dual computation significantly increases computational complexity and inference latency, leading to slower generation speeds and reduced usability. To investigate the impact of various components in the controllable generative models on generation efficiency and quality, we visualize the intermediate results of incorporating conditions at different stages of the model, as shown in Figure 1 . Where the model (a) applies conditions throughout all steps, (b) applies conditions only in the first ten steps, and (c) applies conditions only after the first ten steps. Comparing (a) and (b), the image structure increasingly aligns with the conditional image during the latter stages of the model’s denoising process, indicating lower computational demands at this point. However, comparing (a) and (c) reveals that omitting the conditional image constraint early on results in significant differences in the final output. Additionally, the discrepancies between (b) and (c) underscore the critical influence of when the conditional image is introduced. This suggests that establishing the semantic structure early is essential, though its importance diminishes significantly during the later stages of detail generation. These results show that there are a lot of redundant calculations in controllable generative models, and not every step requires the introduction of conditional calculations. To reduce the computational complexity of the model, recent methods have introduced caching mechanisms. For example, Faster Diffusion ( 12 ) employs temporal-aware optimization, leveraging the gradual evolution of encoder features compared to rapidly changing decoder features across timesteps. It selectively skips encoder computations and reuses cached decoder outputs. DeepCache ( 20 ) targets structural redundancy in denoising paths, exploiting deep-layer feature similarity to bypass recomputation through intelligent caching while preserving shallow-layer updates. T-GATE ( 42 ) accelerates attention mechanisms by caching stabilized cross-attention outputs after convergence, minimizing redundant calculations. However, directly implementing caching in this model may compromise control fidelity, as the control module interacts closely with the generative module, requiring intensive computations. Research ( 10 ) indicates that the generative module’s computational demands surge in later stages, focusing on refining fine-grained details during the final denoising steps. To address this, we propose H ybrid- g rained C aches ( HGC ), which combines block-level caching for stage-wide efficiency with prompt-level caching for granular optimization. First, we introduce a block-level caching strategy to speed up the generation process of the controllable generative model. Specifically, we compute the similarity between block outputs to pinpoint the moment when the structure changes the most, and cache the state at this critical moment to preserve the intermediate representation of the former stage (where the conditional image has the greatest impact). Reusing these cached blocks in subsequent steps can eliminate redundant computations in stages that focus on detail refinement rather than structure redefinition. It reduces computational overhead by avoiding repeated processing of the conditional image in all stages, while ensuring that the final output is consistent with the expected guidance by maintaining early semantic integrity. These operations are used in both the control module and the generative module, but in the generative module, we introduce two control ratios to adjust the cache strength between different modules and denoising stages. Although block-level caching reduces redundant computation by reusing early structural representations, it still requires processing the entire block computation (including the cross-attention layer and multi-layer perceptron (MLP)). Therefore, to further reduce the computational cost, we propose a prompt-level caching strategy that achieves intra-block performance savings by focusing on the cross-attention mechanism. By analyzing the similarities between cross-attention outputs of adjacent blocks, we identify moments of minimal change and cache these states at key points, enabling the network to skip repeated computations without sacrificing image quality. This cue-level cache complements the block-level cache by storing and reusing cross-attention states, reducing the need to recompute these outputs and their associated multi-layer perceptrons (MLPs), and further reducing computational overhead while maintaining early semantic fidelity. We define block-level caching as a coarse-grained cache, and correspondingly, prompt-level caching is defined as a fine-grained cache. These caches, with varying granularities, seamlessly integrate into each stage of the controllable generation process to enhance computational efficiency. Our contributions are threefold: • We propose a coarse-grained cache that adjusts intervals based on feature similarity across steps, reducing overhead in structural synthesis while keeping quality. • We design a fine-grained cache that reuses stable cross-attention maps across denoising steps using temporal similarity to skip redundant steps and keep precision. • Our HGC reduces MACs by approximately 63% across diverse datasets and tasks, achieving comparable quality while significantly boosting model speed. 2 Related Work In this section, we briefly introduce the controllable generative models and the related acceleration methods. 2.1 Controllable Generative Models Ho et al. ( 7 ) introduced the Denoising Diffusion Probabilistic Model (DDPM), laying a robust foundation for iterative denoising in diffusion-based generative models. Dhariwal and Nichol ( 4 ) added classifier guidance to direct sampling toward specific attributes, while Ho et al. ( 9 ) advanced this with classifier-free guidance, embedding conditions directly into the model for enhanced flexibility and sample quality. For spatially conditioned synthesis, Zhang et al. ( 41 ) extended text-to-image diffusion models with ControlNet, enabling precise geometric control, later improved by ControlNet++ ( 11 ) through consistency feedback to minimize artifacts. In text-to-image generation, latent diffusion models like Stable Diffusion ( 27 ) and Imagen ( 28 )

Read the complete text