PatchedServe : A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving
Desen Sun, Zepeng Zhao, Yuke Wang · 2025
Read PatchedServe : A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving by Desen Sun, Zepeng Zhao, Yuke Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09253.txt.
Introduction
Desen Sun, Zepeng Zhao, and Yuke Wang build a serving system for text to image diffusion models in a 2025 paper. Cloud services must answer many image requests at mixed resolutions under strict time promises. Serving whole images blocks batching across requests, and varied sizes break any single cache rule.
Their system works at the level of image patches instead. Patch handling raises throughput on mixed loads, and a patch level cache reuses repeated work across requests. A scheduler that predicts delay online steers patches to meet time promises. Tests show about thirty percent higher promise keeping than the best past system, with image quality intact. The design suits busy creative platforms.
Catalog record
| CREATOR | Desen Sun, Zepeng Zhao, Yuke Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09253
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PatchedServe : A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving Desen Sun, Zepeng Zhao, Yuke Wang Abstract The Text-to-Image (T2I) diffusion model is one of the most popular models in the world. However, serving diffusion models at the entire image level faces several problems, especially when there are multiple candidate resolutions. First, image based serving system prevents requests with different resolutions from batching together. On the other hand, requests with hybrid resolutions also indicate diverse locality features, which makes it hard to apply the same cache policy to all of them. To this end, we propose PatchedServe , A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving that provides a patch-level management strategy to gather hybrid resolution requests into batches. Specifically, PatchedServe incorporates a novel patch-based processing workflow, significantly enhancing throughput for hybrid resolution inputs. Furthermore, PatchedServe designs a patch-level cache reuse policy to fully exploit the redundancy in diffusion. In addition, PatchedServe features an SLO-aware scheduling algorithm with lightweight online latency prediction, achieving higher SLO satisfaction rates. We show that PatchedServe can achieve 30.1 % higher SLO satisfaction compared to SOTA diffusion serving system while not hurt the image quality. 1 Introduction Over recent years, Text-to-Image (T2I) generation tasks have become increasingly popular and have been instrumental for many companies, including Google Imagen [ 40 ] , Adobe Firefly [ 1 ] , and OpenAI DALL·E [ 31 ] . Behind these successful applications, diffusion models have emerged as the dominant paradigm for image generation [ 8 ] , primarily due to their ability to generate images with superior quality. T2I diffusion models synthesize images from Gaussian noises by iteratively denoising, leveraging both ResNet [ 13 ] and Transformer [ 43 ] architectures to create photorealistic images. Despite their remarkable success, serving such models in real-world settings entails significant computational overhead due to several unique characteristics. First, unlike traditional CNN models (e.g., ResNet [ 14 ] and VGG [ 41 ] ) with millisecond-scale latency or modern vision language models (e.g., LLaVA [ 23 ] ) that achieve hundreds of frames per second, mainstream diffusion models typically require several seconds to generate a single image. For instance, it generally takes 3 to 4 seconds for the popular SDXL model [ 36 ] to generate one 512\times 512 image on a powerful NVIDIA H100 GPU). Second, real-world serving scenarios often involve simultaneous requests with varying image resolutions [ 36 , 31 ] , each with different latency requirements. For example, if a new request arrives when another request with a different resolution is being processed on the same GPU, the new request must wait until the previous one completes due to resolution mismatches. This results in prolonged execution times and inefficient GPU utilization. Existing system optimizations for diffusion models primarily consider the simplified uni-resolution input settings and can be broadly categorized into two main approaches. One approach focuses on exploiting the computational parallelism of diffusion models, such as leveraging multiple GPUs or partitioning images into patches [ 20 ] for patch-level parallelism within a single request. However, these methods are limited to handling batches of images with identical resolutions and cannot exploit the opportunity to improve GPU utilization by batching images with misaligned shapes. Another line of research emphasizes harnessing the locality benefits of diffusion models. For instance, block caching [ 47 ] reduces computational complexity by caching intermediate states at each step. However, they can only profile the state change pattern at the granularity of entire images, overlooking the locality benefits at the finer granularity of image patches. With the above insights, we propose a novel approach to optimize diffusion model systems by reorganizing the input at the patch-level granularity , enabling more flexible cache reuse and simplifying mixed-resolution batching. Our key observation is that most of the operations (e.g., Linear and Cross Attention ) in diffusion models focus on “local” (small image patches) rather than “global” (entire image) information. These operations, originally designed for entire images, can be decomposed into mini-operations targeting small image patches. These small patches have the same shape thus we can easily batch them together. Based on this insight, we can reorganize images with different resolutions into patches with uniform shapes, derived from the common factors of their original dimensions, and then batch all patches together to process and schedule them more efficiently. For a simple illustration ( Figure 1 ), there are three diffusion requests with different resolutions. If we serve these three requests directly, we have to process them sequentially due to misaligned input resolutions ( Figure 1 (a)). In such cases, GPUs are more likely to be underutilized since only one one resolution can be processed at a time. Alternatively, if we segment the requests into fine-grained image patches with the same shape, they can be processed in a batched mannar ( Figure 1 (b)). Once processed, the interrelated patches can be concatenated to reconstruct the complete images. With patching, the diffusion serving can exploit abundant locality and parallelism benefits across different levels of compute and memory hierarchies, including inter-patch computational parallelism and intra-patch data locality, to reducing the overall execution latency. Figure 1: Multiple Requests Infer. (a) Serving system without patched inference. It fails to combine them into a single batch due to the diverse resolution. (b) The system that processes requests in patch granularity. Even with three requests in different shapes, it can split them to the same size of patches and better leverage GPU capacity, achieving higher throughput. Despite such potential performance gains from patching, there are still several challenges preventing us from harvesting them. First , dividing an image into patches can lead to the loss of contextual information between them. For example, GroupNorm operations in U-Net-based diffusion models will calculate the mean and variance values of the entire image. Although it is easy to calculate these metrics for individual patches, it is difficult to calculate them for the whole image due to the extensive exchange of partial results across patches. Second , frequent caching operations can introduce severe overhead. To effectively reuse the patch-level cache while preserving image quality, the system must identify which patch outputs are sufficiently similar to cached results from the previous computation. These decisions must be made online before processing each block, and this process is repeated hundreds of times during generation. Even worse, cache management operations (such as insert , delete , query , and re-organize ) are not computationally light, demanding sophisticated design and optimization. Third , diverse SLO requirements add complexity to scheduling. In the real world, the SLO requirement is often 1-5 \times of the model execution latency [ 22 ] , meaning the SLO requirements will always be different when the model latency changes. For instance, if a task with a higher resolution arrives earlier than one with a lower resolution, it is permissible for the larger task to be started later. The more candidate resolutions there are, the more diverse the SLO requirements will be. Such diversity will enlarge the scheduling searching space significantly, makes us hard to find the optimal solution. To address these problems, we propose PatchedServe , A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving ( Figure 10