Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model
Anirud Aggarwal, Abhinav Shrivastava, Matthew Gwilliam · 2025
Read Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model by Anirud Aggarwal, Abhinav Shrivastava, Matthew Gwilliam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15682.txt.
Introduction
Anirud Aggarwal, Abhinav Shrivastava, and Matthew Gwilliam speed up image-making AI with a learned caching plan, in a 2025 paper. Diffusion models make high-quality images but run slowly. Caching parts of the work between steps can help. Older caching rules are rigid and may not fit new models.
Their method uses a genetic search to learn a caching schedule for each model. It needs only a small set of sample prompts. No model weights are changed. The learned schedules trade image quality against speed in fine steps. They also carried over to new image sizes and model variants. The result is faster image generation from off-the-shelf models, with control over the cost.
Catalog record
| CREATOR | Anirud Aggarwal, Abhinav Shrivastava, Matthew Gwilliam |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15682
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model Anirud Aggarwal, Abhinav Shrivastava, Matthew Gwilliam Abstract Diffusion-based image generation models excel at producing high-quality synthetic content, but suffer from slow and computationally expensive inference. Prior work has attempted to mitigate this by caching and reusing features within diffusion transformers across inference steps. These methods, however, often rely on rigid heuristics that result in limited acceleration or poor generalization across architectures. We propose E volutionary C aching to A ccelerate D iffusion models (ECAD), a genetic algorithm that learns efficient, per-model, caching schedules forming a Pareto frontier, using only a small set of calibration prompts. ECAD requires no modifications to network parameters or reference images. It offers significant inference speedups, enables fine-grained control over the quality-latency trade-off, and adapts seamlessly to different diffusion models. Notably, ECAD’s learned schedules can generalize effectively to resolutions and model variants not seen during calibration. We evaluate ECAD on PixArt- α \alpha , PixArt- Σ \Sigma , and FLUX-1.dev using multiple metrics (FID, CLIP, Image Reward) across diverse benchmarks (COCO, MJHQ-30k, PartiPrompts), demonstrating consistent improvements over previous approaches. On PixArt- α \alpha , ECAD identifies a schedule that outperforms the previous state-of-the-art method by 4.47 COCO FID while increasing inference speedup from 2.35x to 2.58x. Our results establish ECAD as a scalable and generalizable approach for accelerating diffusion inference. Our code is available at https://github.com/aniaggarwal/ecad . 1 Introduction Figure 1: We conceptualize diffusion caching as a Pareto optimization problem over image quality and inference time and propose ECAD to discover such Pareto frontiers using a genetic algorithm. Left : performance progression over generations for FLUX-1.dev. Right : example 1024{\times}1024 results with corresponding speedups. Diffusion has emerged as the backbone for state-of-the-art image and video synthesis techniques [ 1 , 2 , 3 , 4 ] . Unlike prior methods involving deep learning, which would train a neural network to generate images in a single forward inference step, diffusion instead involves iterating over a prediction for many (20 to 50) steps [ 5 ] . This process is quite expensive, and many researchers and practitioners try to reduce the latency while preserving, or even improving, the quality [ 6 , 7 , 8 , 9 , 10 ] . Some of these strategies involve training some model that can perform the inference in 1 to 4 steps, particularly with model distillation [ 11 , 9 ] . Other strategies do not train or tune any neural network weights, principally caching [ 6 , 7 , 12 ] , where the diffusion model’s internal features are re-used across steps, allowing that computation to be skipped. We introduce a new conceptual and algorithmic framework for diffusion caching by reframing the problem and replacing existing heuristic-based approaches with a principled, optimization-driven methodology that is generalizable across model architectures. Existing caching methods typically offer a few discrete schedules, each with fixed trade-offs—for example, a 2x speedup with moderate quality loss, and a 3x speedup with greater degradation—without support for intermediate or more aggressive configurations. However, real-world deployments often operate under variable latency or quality constraints, necessitating further flexibility. We instead formulate caching as a multi-objective optimization problem, aiming to discover a smooth Pareto frontier that reveals a wide spectrum of speed-quality trade-offs. We show example frontiers we discover for FLUX.1-dev [ 13 ] in Figure 1 . Such frontiers are very challenging to produce given how caching schedules are currently derived. State-of-the-art approaches [ 8 , 14 , 15 , 16 ] are motivated by heuristics, and key hyperparameters must be carefully hand-tuned by human practitioners based on performance on some set of key metrics. We propose a different paradigm that does not rely on human-defined heuristics or hyperparameters, instead discovering effective caching schedules via genetic algorithm. Our Evolutionary Caching to Accelerate Diffusion models (ECAD) requires two components: (i) some small set of text-only “calibration” prompts and (ii) some metric which computes image quality given a prompt and generated image (we use Image Reward [ 17 ] ). We formulate caching schedules such that the genetic algorithm can automatically discover which features to cache (in terms of blocks and layer types) and when (which timestep). ECAD can be initialized with either random schedules or some set of promising schedules based on prior works [ 8 , 15 ] . Thus, while ECAD presents a different paradigm compared to prior works, it can also build on their valuable findings. ECAD takes these initial schedules and gradually evolves them according to the mating rules of a genetic algorithm, optimizing their “fitness” according to quality and computational complexity (measured in MACs, to be hardware-agnostic). This strategy is extremely flexible. While other methods are entirely designed around whether they cache entire block outputs, intermediate layer outputs (such as the output of an attention layer, or a feedforward layer), or even specific tokens, ours is orthogonal to all of these. We offer a paradigm which can be used to optimize caching schedules according to any well-defined criteria. We instantiate it with our criteria and schedule definitions in Section 3 , but the general principles can be applied to arbitrary criteria and schedules to find Pareto-optimal caching frontiers. For example, we could use other criteria to define fitness, such as human ratings of generated samples. We could also change the caching schedule definitions to be more granular or more general, to focus on certain types of layers, or incorporate heuristics from other methods. Furthermore, while ECAD involves some optimization, since we do not compute any gradients or update any weights, the memory requirements are quite low. Additionally, there are no restrictions on batch size (allowing for use of single, small GPUs that would not be feasible for distillation), and the entire process can happen completely asynchronously. Beyond this, schedules could be optimized for aggressively quantized diffusion models to further improve their acceleration and quality. Figure 1 showcases our method’s strong performance and highlights flexibility across resolutions. Although optimized for FLUX-1.dev at 256{\times}256 , the same schedule applied to 1024{\times}1024 still outperforms SOTA methods in both speed and quality. At 256{\times}256 , ECAD matches or surpasses unaccelerated PixArt- \alpha and FLUX-1.dev baselines with 1.97x and 2.58x latency reductions, respectively. At more aggressive 2.58x and 3.37x settings, quality slightly drops but remains competitive. 2 Related Work 2.1 Diffusion for Image and Video Synthesis Diffusion models predict noise, given noised image inputs, to generate high-quality images [ 2 , 18 , 1 ] and videos [ 3 , 19 , 4 ] . To save time and reduce feature sizes, these computations are typically performed in the latent space [ 20 ] of a pre-trained variational autoencoder [ 21 ] . Although earlier works use U-Net backbones [ 22 ] , more recent methods rely mainly on transformer-based models [ 23 , 24 , 25 , 26 ] , especially Diffusion Transformers (DiTs) [ 25 ] , which dominant the current landscape due to their powerful scaling properties [ 27 , 28 , 4 , 13 ] . Text-conditioning with multimodal models like CLIP [ 29 ] , or extremely powerful text models like T5 [ 30 ] , allows for more granular control over image content [ 31 , 32