CDM-QTA: Quantized Training Acceleration for Efficient LoRA Fine-Tuning of Diffusion Model
Jinming Lu, Minghao She, Wendong Mao, Zhongfeng Wang · 2025
Read CDM-QTA: Quantized Training Acceleration for Efficient LoRA Fine-Tuning of Diffusion Model by Jinming Lu, Minghao She, Wendong Mao, Zhongfeng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-07998.txt.
Introduction
Jinming Lu, Minghao She, Wendong Mao, and Zhongfeng Wang built a training chip for tailoring image models on small devices in a 2025 paper. Fine tuning big diffusion models eats power and time, which strains phones and edge gear. Their chip trains a light add on method in full low precision, cutting memory and power while keeping image quality high.
The design bends its data flow to the odd, shifting tensor shapes the add on method throws up, keeping the hardware busy. Tests showed training up to about one point eight times faster than the base design. Power use per task fell by up to about five and a half times. The authors pitch the chip for on device custom image work.
Catalog record
| CREATOR | Jinming Lu, Minghao She, Wendong Mao, Zhongfeng Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-07998
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CDM-QTA: Quantized Training Acceleration for Efficient LoRA Fine-Tuning of Diffusion Model Thanks: This work was supported in part by the Shenzhen Science and Technology Program 2023A007. Coreesponding author: Wendong Mao, Zhongfeng Wang Jinming Lu, Minghao She, Wendong Mao, Zhongfeng Wang Abstract Fine-tuning large diffusion models for custom applications demands substantial power and time, which poses significant challenges for efficient implementation on mobile devices. In this paper, we develop a novel training accelerator specifically for Low-Rank Adaptation (LoRA) of diffusion models, aiming to streamline the process and reduce computational complexity. By leveraging a fully quantized training scheme for LoRA fine-tuning, we achieve substantial reductions in memory usage and power consumption while maintaining high model fidelity. The proposed accelerator features flexible dataflow, enabling high utilization for irregular and variable tensor shapes during the LoRA process. Experimental results show up to 1.81× training speedup and 5.50× energy efficiency improvements compared to the baseline, with minimal impact on image generation quality. I Introduction Diffusion models have achieved remarkable success in image generation and artistic creation, allowing users to generate high-quality images from simple text prompts. These systems are capable of generating a vast array of objects, styles, and scenes—almost “anything and everything” [ 1 , 2 , 3 , 4 ] . As a versatile class of generative models, diffusion models have demonstrated notable capabilities across a variety of applications, including image super-resolution [ 5 , 6 ] , inpainting [ 7 ] , shape generation [ 8 ] , image-to-image translation [ 9 ] , and molecular conformation generation [ 10 ] . However, despite their broad and general capabilities, users often wish to synthesize specific concepts based on their personal experiences, such as family members, pets or personal items. These concepts are not encountered during the large-scale pre-training procedure. Describing such concepts through text can be cumbersome, and most generative models struggle to reproduce these personal concepts with sufficient fidelity, which has increased demand for model customization [ 11 ] . Fig. 1 : Cross-Attention module in the custom diffusion model. Custom Diffusion [ 11 ] was proposed to enhance existing text-to-image diffusion models using a few user-provided images to incorporate new concepts. The fine-tuned model is then capable of generating new variations with existing concepts. Specifically, a small subset of model weights is identified, namely the key and value mappings from text to latent features in the cross-attention layers, while the rest remain frozen and do not participate in updates. To prevent model forgetting, a small set of real images with similar captions is used as target images. In traditional deep neural networks (DNNs) training, 32-bit single precision floating-point (FP32) has been the default across many DNN training frameworks and hardware systems. Although only 5% of the weight are updated during fine-tuning, the frozen weights still participate in computation in subsequent steps. Therefore, despite the power of diffusion models, their application is limited by the massive number of parameters and computational complexity. For example, running Stable Diffusion [ 2 ] requires 16GB of memory and GPUs with over 10GB of VRAM, which is impractical for most consumer-grade PCs, let alone resource-constrained edge devices. To address the above challenges, we proposed an efficient hardware accelerator for custom diffusion model. Our contributions are summarized as follows. 1. We propose an efficient fine-tuning method based on Low rank adaptation (LoRA) [ 12 ] , designed to expedite the concept fusion process. Subsequently, a quantized training method is developed to reduce computational resources and memory demands significantly, facilitating the implementation of integer calculations during training. 2. We design a flexible hardware accelerator featuring a configurable dataflow that supports both weight stationary (WS) and output stationary (OS) modes. This flexibility allows efficient processing of irregular and small tensor computations in LoRA custom diffusion. 3. Our experimental evaluation shows up to 1.81× training speedup and 5.50× energy efficiency improvement over the baseline architecture. Our design achieves 1.64× and 1.83× and improvements in terms of energy efficiency and area efficiency, respectively, compared to previous work. Fig. 2 : LoRA fine-tuning for Custom Diffusion model. Only weights in pink color are trainable, which accounts for a accounts for a tiny fraction of the entire model. II Algorithm In this section, we introduce our comprehensive compression scheme designed to optimize the performance and efficiency of diffusion models. Our approach consists of two key components: a fine-tuning scheme leveraging Low-Rank Adaptation (LoRA) and a fully quantized scheme. These components are engineered to reduce computational demands while maintaining or improving model accuracy and output quality. This dual strategy streamlines the fine-tuning process, making it feasible to deploy diffusion model in resource-constrained environments. II-A Fine-Tuning Scheme Based on LoRA As shown in Figure 1 , during the fine-tuning process of Custom Diffusion models, a new modifier token, V^{*} is introduced in front of the category name. This fine-tuning primarily optimizes the key and value projection matrices within the cross-attention layers of the diffusion model, alongside the modifier token. The layers involved in optimization are referred to as non-frozen layers , while those that do not participate are termed frozen layers . Consequently, the challenge of compressing the concept fusion process is translated into optimizing these layers using fewer resources. \mathbf{Y}=\mathbf{XW}+\mathbf{XAB}^{T}, (1) To address this, we have implemented a substitution of some non-frozen layers with Low-Rank Adaptation (LoRA). As shown in Figure 2 , this adaptation redefines the original linear transformation in the cross-attention layers as Eq. ( 1 ), where \textbf{X}\in\mathbb{R}^{n\times d_{1}},\textbf{W}\in\mathbb{R}^{d_{1}\times d_{2}},\textbf{A}\in\mathbb{R}^{d_{1}\times r},\textbf{B}\in\mathbb{R}^{d_{2}\times r} , and \textbf{r}\ll min({d_{1},d_{2}}) . Accordingly, the update of the large size weight matrix W is converted to the update of two low-rank matrices A and B . As a result, only 5% of total parameters actively participate in updates, leading to significant savings in computational resources and memory costs. Moreover, given the inherent support for LoRA within the diffusion model framework, replacing parts of the model with LoRA does not result in a substantial loss of accuracy. Fig. 3 : Mixed precision quantization scheme based on LoRA (a) System architecture. (b) WS dataflow. (c) OS dataflow. Fig. 4 : Overview of hardware architecture and dataflow.(a) The hardware architecture of the proposed accelerator. (b) and (c) are the WS and OS dataflows for various computation processes. II-B Fully Quantized Training Scheme Even though the training parameters are significantly reduced after applying customization LoRA fine-tuning for diffusion model, the overall amount of MAC operations and memory consumption of weights and intermediate data are still relatively high. The overall computing graph during training is shown in Fig. 3 , from which we can find that frozen weight still participate in the computation of backward propagation. To further reduce the complexity, we introduce a fully quantized training approach, where weights, activation, and gradients are all quantized into 8-bit integer format (INT8). To ensure the training convergence, a per-tensor quantization scheme is applied to weights, and per-channel/per-column quantization scheme is applied to activations and gradients. The quantization process is written as Eq. 2 . \displaystyle S=\frac{X_{\text{max}}}{2^{q-1}-1}, (2) \displaystyle\widetilde{X}=\text{round}\left(\frac{X}{S}\right)\times S. III Proposed Flexible Hardware Design III-A Hardware Architecture The system architecture is presented in Fig. 4a , which consists of a control module, an N\times N systolic array-based compute module, and SRAM memories.