HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models
Young D. Kwon, Rui Li, Sijia Li, Sourav Bhattacharya, Stylianos I. Venieris · 2025
Read HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models by Young D. Kwon, Rui Li, Sijia Li, Sourav Bhattacharya, Stylianos I. Venieris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04663.txt.
Introduction
Young Kwon and co-authors present a 2025 paper on slimming giant image-making models for small devices. Top text-to-image models hold billions of settings and strain phones and edge gear. The team noticed that the model's blocks play split roles: early ones set the scene's gist, later ones polish texture.
Their method prunes with that split in mind. It cuts the least-needed late blocks, shields the early blocks that carry the gist, and re-teaches the shrunk model with care tuned to each block's sensitivity. The result fits on-device use while keeping image quality close to the full model's.
Catalog record
| CREATOR | Young D. Kwon, Rui Li, Sijia Li, Sourav Bhattacharya, Stylianos I. Venieris |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04663
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models Young D. Kwon, Rui Li, Sijia Li, Sourav Bhattacharya, Stylianos I. Venieris Abstract State-of-the-art text-to-image diffusion models (DMs) achieve remarkable quality, yet their massive parameter scale (8-11B) poses significant challenges for inferences on resource-constrained devices. In this paper, we present HierarchicalPrune , a novel compression framework grounded in a key observation: DM blocks exhibit distinct functional hierarchies, where early blocks establish semantic structures while later blocks handle texture refinements. HierarchicalPrune synergistically combines three techniques: (1) Hierarchical Position Pruning, which identifies and removes less essential later blocks based on position hierarchy; (2) Positional Weight Preservation, which systematically protects early model portions that are essential for semantic structural integrity; and (3) Sensitivity-Guided Distillation, which adjusts knowledge-transfer intensity based on our discovery of block-wise sensitivity variations. As a result, our framework brings billion-scale diffusion models into a range more suitable for on-device inference, while preserving the quality of the output images. Specifically, when combined with INT4 weight quantisation, HierarchicalPrune achieves 77.5-80.4% memory footprint reduction ( e.g. , from 15.8 GB to 3.2 GB) and 27.9-38.0% latency reduction, measured on server and consumer grade GPUs, with the minimum drop of 2.6% in GenEval score and 7% in HPSv2 score compared to the original model. Last but not least, our comprehensive user study with 85 participants demonstrates that HierarchicalPrune maintains perceptual quality comparable to the original model while significantly outperforming prior works. † † footnotetext: *Co-first authors 1 Introduction Diffusion-based text-to-image (T2I) synthesis ( Song et al. 2021 ; Stability AI 2023 ; Lin, Wang, and Yang 2024 ; Lipman et al. 2023 ; Karras et al. 2022 ; Peebles and Xie 2022 ; Lu et al. 2024 ) has emerged as a powerful tool for a wide variety of applications, such as the generation of educational content, creative artwork, and UX/UI design prototyping, intensifying demand for deployable billion-parameter diffusion models (DMs). While recent advances, such as Stable Diffusion 3.5 (SD3.5) ( Esser et al. 2024 ) and FLUX ( Black Forest Labs 2024 ) , significantly outperform previous generations (SDXL ( Podell et al. 2023 ) , SD1.5 ( Rombach et al. 2022 ) , and DALLE-2) in image quality and text alignment, they also come with excessive model sizes (8-11B parameters) and compute demands, limiting the accessibility of such advanced models. At the same time, despite quantitative metrics ( e.g. , GenEval ( Ghosh, Hajishirzi, and Schmidt 2023 ) and DPG-Bench ( Hu et al. 2024b ) ) suggesting smaller models (approx. 2B parameters) like SANA-Sprint ( Chen et al. 2025b ) perform better than their large-scale counterparts, user-based evaluations on the Artificial Analysis Leaderboard 1 1 1 https://artificialanalysis.ai/text-to-image/arena?tab=leaderboard-text reveal a significant gap in the perceived quality between compact models and their larger counterparts, which quantitative metrics fail to capture. As such, there is a longstanding need for deploying large models even in resource-constrained settings in order to provide users with high-quality T2I capabilities. Figure 1: HierarchicalPrune achieves 79.5% memory reduction (left) while maintaining image quality. User study (right) demonstrates minimal quality drop (4.8-5.3%), contrary to the excessive degradation (11.1-52.2%) of prior methods. Figure 2: High-resolution image samples generated by our pruned/distilled model using HierarchicalPrune , showcasing its superior visual quality across various visual styles, precisely following text prompts, and preserving the ability to draw typography. Nonetheless, this endeavour constitutes a very challenging task due to the high memory and compute intensity ( Li et al. 2025 ) of large DMs, thereby often requiring cloud solutions equipped with high-end GPUs and a minimum of 80GB VRAM. Concurrently, most of the renowned recent releases of foundational DMs, including SD3.5 ( Esser et al. 2024 ) , FLUX ( Black Forest Labs 2024 ) , and Seedream 3.0 ( ByteDance 2025 ) , are built upon multi-modal diffusion Transformer (MMDiT) backbones ( Esser et al. 2024 ) instead of U-Net, revealing new compression opportunities (Section 2.1 ). At the moment, existing efforts to improve efficiency face important limitations. Firstly, sampling step reduction ( Li et al. 2023 ; Stability AI 2023 ; Sauer et al. 2023 ) and efficient operator design ( Xie et al. 2025 ; Dao et al. 2022 ) are tailored to improving the speed of DMs rather than reducing memory requirements, leaving the important task of DM deployment in memory-constrained devices unresolved. Secondly, existing depth-pruning methods ( Lee et al. 2024 ; Kim et al. 2024a ; Fang et al. 2024 ) show promising results in both memory and computation reduction ( Kim et al. 2024a ; Fang et al. 2024 ) , outperforming width pruning ( Fang, Ma, and Wang 2023 ; Castells et al. 2024a ) , but face critical scalability challenges. While they achieve reasonable compression on U-Net-based small DMs (2.6B or less), they fail to compress large-scale, state-of-the-art (SOTA) DMs such as SD3.5 Large (8B) and FLUX (11B) (experiencing significant degradation at 20-30% memory reduction as demonstrated in Table 1 ). The full-block-removal methods employed therein cannot capture the fine-grained impact of the subcomponents within each block, and more importantly, the different roles of blocks in different positions across the network’s hierarchy, leading to excessive performance drop at high pruning ratios. Lastly, orthogonal to pruning, reduced-precision computation ( He et al. 2023 ; Li et al. 2025 ; Wang et al. 2024 ) has been proven effective, but it has not been combined with prior block-removal DM compression methods. By observing the limitations of prior work, this paper identifies a novel insight for DMs that consist of MMDiT blocks: Such DMs form a two-fold hierarchy, spanning the inter- and intra-block levels. Inter-block hierarchy reflects the contribution of different blocks to disparate aspects of the output image ( e.g. semantic structure, finer visual details), which is determined by their position in the overall architecture. Intra-block hierarchy highlights the subcomponents that compose each MMDiT block and their diverse patterns of importance to the overall quality. Building on this new viewpoint, we propose HierarchicalPrune , a principled compression methodology that overcomes the limitations of existing methods using three hierarchy-informed techniques (Fig. 3 ). First, we introduce Hierarchical Position Pruning (HPP) , a method that leverages our empirical insight that later MMDiT blocks contribute less to fundamental image structure, by strategically maintaining early blocks that form core image structures while pruning later blocks that primarily handle refinements. Second, we incorporate Positional Weight Preservation (PWP) , which freezes the non-pruned and earlier portions of the model during the distillation process. This approach maintains the integrity of early blocks, which are essential for image formation, while allowing later, less critical blocks to be updated. Finally, we propose Sensitivity-Guided Distillation (SGDistill) , which operates with a counterintuitive yet effective principle: blocks with higher importance are also more sensitive to change. Our analysis reveals that in aggressive pruning settings, attempting to update these highly important blocks often proves detrimental to model performance. As such, we enforce inverse distillation weights—assigning minimal or zero update weights to the most important blocks, while concentrating updates on less sensitive components. We extensively benchmark HierarchicalPrune on both server and desktop-grade GPUs, demonstrating superior results over SOTA methods. Our contributions include: • We identify a dual hierarchical structure in MMDiT DMs: an inter-block hierarchy