Gradient Weight-normalized Low-rank Projection for Efficient LLM Training

Jia-Hong Huang, Yixian Shen, Hongyi Zhu, Stevan Rudinac, Evangelos Kanoulas · 2024

Read Gradient Weight-normalized Low-rank Projection for Efficient LLM Training by Jia-Hong Huang, Yixian Shen, Hongyi Zhu, Stevan Rudinac, Evangelos Kanoulas (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19616.txt.

Introduction

Jia-Hong Huang and four co-authors trim the memory cost of training large language models with a method called GradNormLoRP. Full fine-tuning is powerful but hungry, while cheaper parameter-efficient methods often lag behind and still waste memory. Their approach attacks both weights and gradients at once.

The method normalizes weight matrices to improve how gradients behave during optimization. It then projects weights and gradients into low-rank form, shrinking what must be stored. In eight-bit form it cut optimizer memory by up to about ninety percent in experiments. That saving let a seven-billion-parameter model pretrain on a consumer graphics card. Fine-tuning results also beat earlier low-rank methods, with no added inference cost.

Catalog record

CREATORJia-Hong Huang, Yixian Shen, Hongyi Zhu, Stevan Rudinac, Evangelos Kanoulas
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-19616

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradient Weight-normalized Low-rank Projection for Efficient LLM Training Jia-Hong Huang, Yixian Shen, Hongyi Zhu, Stevan Rudinac, Evangelos Kanoulas Abstract Large Language Models (LLMs) have shown remarkable performance across various tasks, but the escalating demands on computational resources pose significant challenges, particularly in the extensive utilization of full fine-tuning for downstream tasks. To address this, parameter-efficient fine-tuning (PEFT) methods have been developed, but they often underperform compared to full fine-tuning and struggle with memory efficiency. In this work, we introduce Gradient Weight-Normalized Low-Rank Projection (GradNormLoRP), a novel approach that enhances both parameter and memory efficiency while maintaining comparable performance to full fine-tuning. GradNormLoRP normalizes the weight matrix to improve gradient conditioning, facilitating better convergence during optimization. Additionally, it applies low-rank approximations to the weight and gradient matrices, significantly reducing memory usage during training. Extensive experiments demonstrate that our 8-bit GradNormLoRP reduces optimizer memory usage by up to 89.5% and enables the pre-training of large LLMs, such as LLaMA 7B, on consumer-level GPUs like the NVIDIA RTX 4090, without additional inference costs. Moreover, GradNormLoRP outperforms existing low-rank methods in fine-tuning tasks. For instance, when fine-tuning the RoBERTa model on all GLUE tasks with a rank of 8, GradNormLoRP achieves an average score of 80.65, surpassing LoRA’s score of 79.23. These results underscore GradNormLoRP as a promising alternative for efficient LLM pre-training and fine-tuning. Source code: https://github.com/Jhhuangkay/Gradient-Weight-normalized-Low-rank-Projection-for-Efficient-LLM-Training University of Amsterdam \dagger Authors contributed equally to this work [email protected], [email protected], [email protected], [email protected], [email protected] Introduction Large Language Models (LLMs) pre-trained on extensive datasets have demonstrated exceptional effectiveness across various domains ( Devlin et al. 2019 ; Liu et al. 2019 ; He et al. 2022 ; Xie et al. 2022 ; Baevski et al. 2020 ; Lu et al. 2019 ; Tan and Bansal 2019 ) . As time progresses, open-source LLMs have consistently improved in their capabilities, accompanied by a striking increase in the scale of pre-trained models ( Raffel et al. 2020a ; Zhang et al. 2022 ; Le Scao et al. 2023 ; Touvron et al. 2023 ; Tay et al. 2023 ) . Consequently, employing full fine-tuning, where all learnable parameters of a pre-trained model are updated for performing downstream tasks, poses unparalleled challenges despite its track record of delivering numerous state-of-the-art results. These challenges primarily stem from the escalating demands on computational resources. Algorithm 1 Our proposed GradNormLoRP 0: Weight matrix \mathcal{W} 0: Updated weight matrix \mathcal{W}_{\text{updated}} 1: Normalize each column weight vector of \mathcal{W} to get \mathcal{W}_{\text{norm}} 2: Apply LoRA with two low-rank matrices I and J to reformulate \mathcal{W}_{\text{norm}} 3: Initialize two sets of low-rank projection matrices (U_{I},V_{I}) and (U_{J},V_{J}) 4: for i=1 to N do 5: Compute gradient matrices \mathcal{Z}_{I} and \mathcal{H}_{J} based on I and J 6: Project \mathcal{Z}_{I} and \mathcal{H}_{J} using (U_{I},V_{I}) and (U_{J},V_{J}) 7: if i is a multiple of 250 then 8: Update (U_{I},V_{I}) and (U_{J},V_{J}) 9: end if 10: end for 11: return \mathcal{W}_{\text{updated}}(\mathcal{W}_{\text{norm}} , \mathcal{Z}_{I} , (U_{I},V_{I}) , \mathcal{H}_{J} , (U_{J},V_{J})) To tackle the aforementioned challenge, researchers have developed parameter-efficient fine-tuning (PEFT) techniques ( Houlsby et al. 2019 ; Hu et al. 2022 ; Lialin et al. 2023 ; Liu et al. 2024 ; Kopiczko, Blankevoort, and Asano 2024 ) . These methods are tailored to update only a small amount of task-specific parameters while leaving the majority of the model’s parameters unchanged. Among these techniques, low-rank approximation-based approaches utilize low-rank matrices to approximate weight changes during training, achieving both parameter and memory efficiency without requiring additional trainable subnetworks to be added to the original model architecture. Despite their advantages, low-rank-based methods often underperform compared to full-rank fine-tuning ( Hu et al. 2022 ; Lialin et al. 2023 ; Liu et al. 2024 ) . This performance gap is typically attributed to the reduced number of trainable parameters, but other underlying factors, such as altered gradient dynamics due to reparameterization, also play a significant role. In Figure 2 of our Appendix 1 , we observe that the gradient descent process can become neither smooth nor stable when fine-tuning LLMs in an unnormalized subspace. This instability arises from conducting gradient descent on an incomparable scale, where some values are excessively large or small. Such numerical instability can lead to overflow or underflow during computations, negatively impacting the optimization process and resulting in suboptimal performance. To mitigate this problem, we propose our method, Gradient Weight-Normalized Low-Rank Projection (GradNormLoRP). This approach effectively enhances both parameter and memory efficiency. GradNormLoRP improves parameter efficiency by incorporating a weight matrix normalization process that represents each column vector of the weight matrix as the product of its magnitude and unit vector. This normalization enhances gradient conditioning and facilitates better convergence during optimization. In addition to improving parameter efficiency, GradNormLoRP addresses memory efficiency while maintaining performance comparable to full fine-tuning without introducing additional inference burden. Although training LLMs in a normalized subspace enhances convergence during optimization, existing PEFT methods ( Hu et al. 2022 ; Liu et al. 2024 ; Houlsby et al. 2019 ; Pfeiffer et al. 2020 ; Li and Liang 2021 ; Lester, Al-Rfou, and Constant 2021 ; Liu et al. 2022 ) still face limitations in reducing GPU memory usage. Specifically, these methods rely on caching intermediate activations during the forward pass to compute gradients, which remains a significant memory overhead due to the standard backpropagation process. This inefficiency poses difficulties for training LLMs on a single consumer-level GPU, such as the NVIDIA RTX 4090 with 24GB of memory. To address the memory efficiency issue, GradNormLoRP applies a low-rank approximation technique to both the normalized weight matrix and its corresponding gradient matrix. This process involves reformulating the normalized weight matrix as the sum of a fixed pre-trained weight matrix and the product of two low-rank matrices. It also requires computing two sets of low-rank projection matrices to project the gradient matrices derived from these low-rank matrices. These projection matrices are updated periodically, e.g., every 250 iterations, to ensure minimal additional computational overhead over time. Exploiting this technique, our proposed GradNormLoRP achieves both memory and parameter efficiency during training while further enhancing the convergence process of optimization in the normalized subspace. We conduct extensive experiments to demonstrate the effectiveness of our proposed GradNormLoRP in both LLM pre-training and fine-tuning, leveraging the C4 dataset ( Raffel et al. 2020b ) and the GLUE benchmark ( Wang et al. 2019 ) . GradNormLoRP significantly reduces memory usage in optimizer states by up to 89.5%, while preserving efficiency and performance during pre-training on the LLaMA 7B ( Touvron et al. 2023 ) architecture with the C4 dataset, comprising up to 10.2 billion tokens. Furthermore, our 8-bit GradNormLoRP achieves additional reductions, cutting optimizer memory by up to 83.7% and total training memory by 65.2% compared to a BF16 baseline. Remarkably, we demonstrate the feasibility of pre-training the LLaMA 7B model on consumer-level GPUs with 24GB memory, such as the NVIDIA RTX 4090, without necessitating strategies like model parallelism, offloading, or checkpointing. In the realm of fine-tuning pre-trained LLMs on GLUE benchmarks, GradNormLoRP proves superior to existing low-rank methods. For example, when fine-tuning the RoBERTaBase model ( Liu et al. 2019 ) on GLUE tasks with a rank of 8, GradNormLoRP attains an average score of 80.65, outpacing LoRA, which achieves

Read the complete text