Memory-Efficient Fine-Tuning Diffusion Transformers via Dynamic Patch Sampling and Block Skipping

Sunghyun Park, Jeongho KimHyoungwoo ParkDebasmit DasSungrack YunMunawar HayatJaegul ChooFatih Porikli, Seokeon Choi, Qualcomm AI Research KAIST {sunpar, jeonghok, hwoopark, debadas, sungrack, fporikli, seokchoi}@[email protected] · 2026

Read Memory-Efficient Fine-Tuning Diffusion Transformers via Dynamic Patch Sampling and Block Skipping by Sunghyun Park, Jeongho KimHyoungwoo ParkDebasmit DasSungrack YunMunawar HayatJaegul ChooFatih Porikli, Seokeon Choi, Qualcomm AI Research KAIST {sunpar, jeonghok, hwoopark, debadas, sungrack, fporikli, seokchoi}@[email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20755.txt.

Introduction

Park and co-authors cut the memory cost of fine tuning diffusion transformers for image generation. Full fine tuning needs more memory than many devices hold. Their DiT-BlockSkip method samples patches with sizes tied to the diffusion step. Cropped patches shrink to a fixed low resolution. The method also skips less vital blocks during training. Cross attention masks help pick which blocks matter most. Tests kept personalization quality close to full tuning. Memory use fell a lot, moving toward on device use.

Catalog record

CREATORSunghyun Park, Jeongho KimHyoungwoo ParkDebasmit DasSungrack YunMunawar HayatJaegul ChooFatih Porikli, Seokeon Choi, Qualcomm AI Research KAIST {sunpar, jeonghok, hwoopark, debadas, sungrack, fporikli, seokchoi}@[email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58499
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Memory-Efficient Fine-Tuning Diffusion Transformers via Dynamic Patch Sampling and Block Skipping Sunghyun Park, Jeongho KimHyoungwoo ParkDebasmit DasSungrack YunMunawar HayatJaegul ChooFatih Porikli, Seokeon Choi, Qualcomm AI Research KAIST {sunpar, jeonghok, hwoopark, debadas, sungrack, fporikli, seokchoi}@[email protected] Abstract Diffusion Transformers (DiTs) have significantly enhanced text-to-image (T2I) generation quality, enabling high-quality personalized content creation. However, fine-tuning these models requires substantial computational complexity and memory, limiting practical deployment under resource constraints. To tackle these challenges, we propose a memory-efficient fine-tuning framework called DiT-BlockSkip, integrating timestep-aware dynamic patch sampling and block skipping by precomputing residual features. Our dynamic patch sampling strategy adjusts patch sizes based on the diffusion timestep, then resizes the cropped patches to a fixed lower resolution. This approach reduces forward & backward memory usage while allowing the model to capture global structures at higher timesteps and fine-grained details at lower timesteps. The block skipping mechanism selectively fine-tunes essential transformer blocks and precomputes residual features for the skipped blocks, significantly reducing training memory. To identify vital blocks for personalization, we introduce a block selection strategy based on cross-attention masking. Evaluations demonstrate that our approach achieves competitive personalization performance qualitatively and quantitatively, while reducing memory usage substantially, moving toward on-device feasibility ( e.g. , smartphones, IoT devices) for large-scale diffusion transformers. † † ∗ These two authors contributed equally to this work. † † † Corresponding author. † † ‡ Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. 1 Introduction With the advent of large-scale text-to-image (T2I) generation frameworks, such as Stable Diffusion Rombach et al. 2022 , personalized content creation has surged, driving applications such as avatar generation and customized imagery Gal et al. 2023 ; Ruiz et al. 2023 ; Kumari et al. 2023 ; Ruiz et al. 2024 ; Wei et al. 2023 ; Shi et al. 2024 . A common approach to personalization involves fine-tuning pre-trained T2I models using a few user-provided reference images Ruiz et al. 2023 ; Gal et al. 2023 ; Kumari et al. 2023 . However, this fine-tuning process incurs substantial computational and memory overhead, especially large-scale models, limiting practical deployment under resource constraints. Figure 1 : Memory consumption analysis during fine-tuning DiT ( i.e. , FLUX). (a) We propose DiT-BlockSkip including patch sampling and block skipping, which reducing the training memory usage significantly. (b) Comparison with baselines and DiT-BlockSkip in aspects of GPU memory usage and subject & text fidelity. To mitigate these issues, parameter-Efficient Fine-Tuning (PEFT) methods, including LoRA Hu et al. 2022 and its variants Liu et al. 2024 ; Kopiczko et al. 2023 ; Bhardwaj et al. 2024 ; Zhang et al. 2023 , freeze the original base model weights and only update small amounts of parameters. However, these methods still require full backpropagation through the entire network, resulting in substantial memory overhead due to the excessive number of model parameters and activations involved. This remains a challenge even with quantization Dettmers et al. 2023 ; Xu et al. 2023 ; Kim et al. 2023 ; Ryu et al. 2024 , which can further compromise precision. Recently, several approaches have adopted gradient-free or zeroth-order optimization strategies Fei et al. 2023 ; Seo et al. 2025 to fine-tune models without relying on backpropagation, thereby significantly reducing memory consumption during fine-tuning. However, these approaches often suffer from instability during optimization and require a large number of training iterations to converge Seo et al. 2025 . With the emergence of diffusion transformer (DiT) Peebles and Xie 2023 -based T2I models Esser et al. 2024 ; Labs 2024 ; Chen et al. 2024a ; Xie et al. 2024 , recent architectures have dramatically increased in depth and capacity, enabling the generation of high-resolution images. However, this advancement comes at the cost of significantly higher memory consumption during fine-tuning, posing challenges for practical deployment, particularly in resource-constrained scenarios. Furthermore, the existing efficient fine-tuning techniques Cho et al. 2024 ; Fei et al. 2023 ; Ryu et al. 2024 ; Seo et al. 2025 have validated on U-Net-based T2I models, leaving DiT-based architectures relatively underexplored. Therefore, achieving effective personalization with DiT-based T2I models necessitates further reductions in memory consumption. To address these challenges, we propose a novel scheme, called DiT-BlockSkip , that significantly reduces the training memory overhead while preserving output quality (Fig. 1 ). Our method consists of two key components: (1) dynamic patch sampling and (2) block skipping with residual feature precomputation. The first component reduces forward & backward memory significantly by using on low-resolution patch images, wherein the patch size is dynamically adjusted based on the diffusion timestep. Here, we first crop the high-resolution images based on the adjusted patch sizes, and then we resize these patches of varying sizes into a fixed low-resolution size (Sec. 3.1 ). The core intuition is that higher timesteps are associated with learning global structure, while lower timesteps are more effective for capturing fine-grained details Yi et al. 2024 ; Saichandran et al. 2025 . By aligning crop ratios with these characteristics, the model can effectively learn both structural and detailed information even from low-resolution inputs, mimicking the representational benefits of high-resolution training. The second component, block skipping with residual feature precomputation, selectively updates only a subset of essential transformer blocks for personalization during fine-tuning (Sec. 3.2 ). For skipping blocks, precomputed residual feature maps are reused, allowing the model to bypass unnecessary forward and backward computations. This strategy is motivated by the observation that full fine-tuning of DiT is often redundant, and learning can be preserved by focusing on a strategically chosen subset of critical blocks. Moreover, we introduce a block selection strategy based on cross-attention masking to identify critical blocks for personalization. Together, the proposed methods significantly reduce the training memory usage while maintaining the personalization performance. Our contributions are summarized as follows: • To reduce the training memory, we introduce a patch sampling strategy that dynamically adjusts the patch sizes based on diffusion timesteps, learning both global structures and fine-grained details from low-resolution images. • We propose a method that selectively fine-tunes essential transformer blocks for personalization while utilizing pre-extracted residual features for the skipped blocks. This technique maintains personalization quality while minimizing computational overhead. • The experiments demonstrate that our method produces competitive performance while significantly reducing training memory usage. 2 Related Work Figure 2 : Overview of the proposed method. (a) The dynamic patch sampling applies different patch sizes for each diffusion timestep, enabling the model to learn both global structure and fine-grained details depending on the noise level. Cropped patches of various sizes are resized to the same fixed resolution ( s_{min}\times s_{min} ) before being fed into the model. (b) Block skipping with residual features for DiT stores the \Delta value between the input and output of the consecutive skipped blocks in the precomputation stage. During fine-tuning, these residual feature is added to the input of the skipped blocks, while the remaining blocks are updated end-to-end. Personalization of Diffusion Models. Personalization in diffusion models aims to adapt pre-trained models to synthesize images of user-defined concepts using reference images. Training-based methods Ruiz et al. 2023 ; Gal et al. 2023 ; Kumari et al. 2023 ; Huang et al. 2024 ; Ruiz et al. 2024 ; Park et al. 2025 typically fine-tune

Read the complete text