LowDiff: Efficient Diffusion Sampling with Low-Resolution Condition
Jiuyi Xu, Qing Jin, Meida Chen, Andrew Feng, Yang Sui, Yangming Shi · 2025
Read LowDiff: Efficient Diffusion Sampling with Low-Resolution Condition by Jiuyi Xu, Qing Jin, Meida Chen, Andrew Feng, Yang Sui, Yangming Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15342.txt.
Introduction
Jiuyi Xu and five co-authors accelerate diffusion image generation by starting small in a 2025 computer vision paper. Diffusion models generate superb images slowly, and most speedups shrink the model or cut denoising steps. LowDiff instead exploits resolution: early generation happens cheaply at low resolution, where mistakes cost little, and detail arrives in a cascade.
A single unified model refines images progressively from low to target resolution, so far fewer expensive high-resolution sampling steps are needed. The framework applies to diffusion models in both pixel space and latent space. Across experiments, image quality matches or beats the standard pipeline at a fraction of the high-resolution compute. Resolution scheduling becomes another dial, beside step count, for trading speed against fidelity.
Catalog record
| CREATOR | Jiuyi Xu, Qing Jin, Meida Chen, Andrew Feng, Yang Sui, Yangming Shi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15342
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LowDiff: Efficient Diffusion Sampling with Low-Resolution Condition Jiuyi Xu, Qing Jin, Meida Chen, Andrew Feng, Yang Sui, Yangming Shi Abstract Diffusion models have achieved remarkable success in image generation but their practical application is often hindered by the slow sampling speed. Prior efforts of improving efficiency primarily focus on compressing models or reducing the total number of denoising steps, largely neglecting the possibility to leverage multiple input resolutions in the generation process. In this work, we propose LowDiff, a novel and efficient diffusion framework based on a cascaded approach by generating increasingly higher resolution outputs. Besides, LowDiff employs a unified model to progressively refine images from low resolution to the desired resolution. With the proposed architecture design and generation techniques, we achieve comparable or even superior performance with much fewer high-resolution sampling steps. LowDiff is applicable to diffusion models in both pixel space and latent space. Extensive experiments on both conditional and unconditional generation tasks across CIFAR-10, FFHQ and ImageNet demonstrate the effectiveness and generality of our method. Results show over 50% throughput improvement across all datasets and settings while maintaining comparable or better quality. On unconditional CIFAR-10, LowDiff achieves an FID of 2.11 and IS of 9.87, while on conditional CIFAR-10, an FID of 1.94 and IS of 10.03. On FFHQ 64×64, LowDiff achieves an FID of 2.43, and on ImageNet 256×256, LowDiff built on LightningDiT-B/1 produces high-quality samples with a FID of 4.00 and an IS of 195.06, together with substantial efficiency gains. 1 Introduction High-fidelity image generation has become a reality thanks to powerful diffusion models, which have set new benchmarks across a wide range of generative tasks baldridge2024imagen ; betker2023improving ; brooks2024video ; esser2024scaling ; karras2024analyzing Although early research mainly focuses on improving generation quality, there is a growing emphasis on accelerating the sampling process to make diffusion models more practical for real-world applications, especially in scenarios with limited computational resources or strict efficiency requirements li2024svdqunat ; li2023q ; li2023snapfusion ; shang2023post ; sui2024bitsfusion ; wu2024snapgen . As image generation scales to higher resolutions and larger datasets, the demand for efficient training and inference has become increasingly critical. To enable efficient diffusion models, previous work has primarily explored three strategies: model compression, denoising step reduction, and lower-dimension generation. First, various model compression techniques, including network pruning fang2023structuralpruningdiffusionmodels ; wan2025pruning , efficient architectural design yang2023diffusion ; zhu2024dig ; phung2023wavelet ; li2023efficient , neural architecture search tang2023lightweight ; li2023snapfusion , and quantization sui2024bitsfusion ; li2023q , have been investigated to reduce the size of diffusion models and the latency per function evaluation. Second, denoising step reduction targets the number of function evaluations (NFEs), which can range from tens to thousands and is a major bottleneck in sampling efficiency. By incorporating techniques including knowledge distillation luhman2021knowledge ; zhang2024accelerating , progressive distillation salimans2022progressive ; meng2023distillation ; li2023snapfusion , and auxiliary networks or loss functions wang2022diffusion ; xiao2021tackling ; yin2024one ; yin2024improved , diffusion models can be accelerated while maintaining generation quality, often by allowing larger denoising step size. Figure 1 : Sample quality vs. effective number of function evalutions. We compare different diffusion models that are trained from scratch on CIFAR10 dataset, and measure the generation quality by FID ( \downarrow ). Our model provides comparable output quality as the previous best results and reduce the required computing efforts by more than 30%. The third approach, lower-dimension generation , follows a divide-and-conquer strategy by breaking down the complex generation task into simpler sub-tasks. For instance, Stable Diffusion rombach2022high integrates a variational autoencoder (VAE) and a diffusion model by applying the diffusion process in latent space and decoding the resulting latent variables into high-quality images. Recent work li2024return also uses intermediate features generated by diffusion models as conditional inputs to enhance subsequent stages of generation. Beyond improving efficiency, some other works also adopt similar hierarchical ideas to improve the quality of generated images ho2022cascaded ; atzmon2024edify ; gu2023matryoshka . However, the potential of leveraging low-resolution generation within these frameworks to improve sampling efficiency remains underexplored. From the perspective of low-resolution, we investigate an alternative approach towards efficient diffusion models by adopting a cascaded generation approach ho2022cascaded , where higher-resolution outputs are conditioned on progressively generated lower-resolution images li2024return , and we aim to alleviate the trade-off between sampling speed and generation quality. Rather than directly generating images of resolution 32 or 64, our method starts with an even lower resolution (e.g., 8 \times 8) and incrementally synthesizes higher-resolution images conditioned on preceding outputs. By conditioning the generation at each stage on lower-resolution images, we are able to reduce the number of denoising steps required for high-resolution outputs, an important consideration given that the computational cost of diffusion models scales quadratically with input resolution. To further enhance efficiency, we propose a unified model architecture in which the diffusion modules across resolutions share the majority of their weights. Only lightweight, resolution-specific input/output adapters and resolution-aware embedding layers are introduced. This design minimizes architectural overhead and preserves compatibility with standard diffusion models including both UNet-based and Transformer-based architectures. Importantly, unlike distillation-based acceleration techniques, our method is trained end-to-end from scratch, without auxiliary models or loss terms, simplifying the training pipeline and improving stability. Our contribution is summarized as the following: • Multi-Resolution Efficient DM. We propose a multi-resolution cascaded diffusion framework that initiates generation from low-resolution inputs to enable efficient diffusion-based image synthesis. Conditioned on the generated low-resolution images, the required number of steps for high-resolution generation is much reduced without impairing the final output quality. • End-to-end Unified DM. We design an end-to-end unified model to handle diffusion generation across multiple resolutions, where the low-resolution sub-network is reused both for low-resolution generation and as sub-blocks for downsampled features in high-resolution model, thus reducing both memory usage and model size. • Improved Generation Quality-Efficiency Trade-off. Our model achieves comparable output quality for several image datasets while improving the throughput performance by more than 50% . 2 Related Works Efficient Diffusion Models. Diffusion models are typically bulky in size and require a large amount of denoising steps to generate one output. To reduce the size and memory requirement of diffusion models, a structural pruning technique is proposed by identifying important weights according to the informative gradients fang2023structuralpruningdiffusionmodels . A progressive soft pruning strategy is investigated based on the gradient flow of the energy function wan2025pruning . To quantize diffusion models, Q-Diffusion li2023q applies post-training quantization techniques with timestep-aware calibration and shortcut quantization split to shrink the model weights to 4-bit. SVDQuant li2024svdqunat quantizes both weights and activations to 4-bit by outlier redistribution. BitsFusion sui2024bitsfusion further reduces the model size below 2-bit by weight quantization-aware training, where during training, time steps are sampled according to the corresponding quantization error of each timestep. To reduce the number of function evaluations (NFEs), distillation technique to progressively reduce the required number of denoising steps is proposed in salimans2022progressive . SnapFusion li2023snapfusion further proposes a CFG-aware step distillation method to take into account the effect of classifier-free guidance, thereby improves the generate quality. Adversarial loss is also introduced to combine the advantages of diffusion models and GAN models in xiao2021tackling ; wang2022diffusion . Meanwhile, the serial studies of DMD yin2024one ; yin2024improved propose to use distribution matching distillation to help