Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention

Xingtai Lv, Ning Ding, Kaiyan Zhang, Ermo Hua, Ganqu Cui, Bowen Zhou · 2024

Read Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention by Xingtai Lv, Ning Ding, Kaiyan Zhang, Ermo Hua, Ganqu Cui, Bowen Zhou (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02063.txt.

Description

Abstract Improving the effectiveness and efficiency of large language models (LLMs) simultaneously is a critical yet challenging research goal. In this paper, we find that low-rank pre-training, normally considered as efficient methods that will compromise performance, can be scalably effective when reduced parameters are precisely targeted.

Catalog record

CREATORXingtai Lv, Ning Ding, Kaiyan Zhang, Ermo Hua, Ganqu Cui, Bowen Zhou
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02063

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention Xingtai Lv, Ning Ding, Kaiyan Zhang, Ermo Hua, Ganqu Cui, Bowen Zhou Abstract Improving the effectiveness and efficiency of large language models (LLMs) simultaneously is a critical yet challenging research goal. In this paper, we find that low-rank pre-training, normally considered as efficient methods that will compromise performance, can be scalably effective when reduced parameters are precisely targeted. Specifically, applying the low-dimensional module only to the attention layer – resolves this issue and enhances both effectiveness and efficiency. We refer to this structure as Low-dimensional Projected Attention (LPA) and provide an explanatory analysis. Through extensive experimentation at parameter scales of 130M, 370M, and scaling up to 3B, we have validated the effectiveness and scalability of LPA. Our results show that LPA model can save up to 12.4% in time while achieving an approximate 5% improvement in test perplexity (ppl) and on downstream tasks compared with the vanilla Transformer. 1 Introduction Improving large language models’ (LLMs) Bommasani et al. (2021) ; Han et al. (2021) ; Brown et al. (2020) ; Touvron et al. (2023) ; Zhou and Ding (2024) effectiveness and efficiency simultaneously presents challenges due to inherent trade-offs, which remains a critical research goal in the research field. Among series methods proposed to alleviate this issue, parameter-efficient fine-tuning Houlsby et al. (2019) ; Li and Liang (2021) ; Zaken et al. (2021) ; Ding et al. (2023b) offer valuable insights. Notably, low-rank or low-dimension techniques such as LoRA Hu et al. (2021) demonstrate on-par or even enhanced performance over traditional full-parameter fine-tuning with reduced computational resources. Intuitively, besides the fine-tuning phase, adapting LoRA’s principles to the pre-training phase through low-rank decomposition is both viable and promising, which can yield substantial benefits if effectiveness is maintained. However, existing studies have found that the direct low-rank pre-training often compromises the effectiveness. To reduce such effects, strategies such as iteratively accumulating low-rank updates Lialin et al. (2023) or integrating low-rank decomposition directly into the gradient Zhao et al. (2024) have been suggested. Whether it’s the original LoRA or these improved methods, they all involve performing low-rank decomposition and updates on "amounts of change" (weights or gradients), and do not reduce the number of parameters in the model itself, which face obstacles in maintaining efficiency during subsequent inference and fine-tuning stages. Therefore, an ideal scenario would be permanently reducing the number of parameters (computational load) through efficient methods, without compromising or even enhancing the performance of pre-trained models. To achieve this goal, is it feasible to directly perform low-rank decomposition on the matrices in the model itself, rather than on the changes? Current limited research suggests that existing low-rank pre-training methods experience performance losses and uncertainties Lialin et al. (2023) ; Zhao et al. (2024) , with even fewer studies exploring more direct approaches. However, in this paper, we demonstrate that such direct low-rank pre-training is feasible, provided that the parameters to be reduced are more precisely targeted . Specifically, we describe the reduction of parameters as replacing the original matrices with low-dimensional modules. We find that using low-dimensional modules in the feed-forward neural (FFN) layers or across all layers negatively impacts the model’s effectiveness. However, we observe that employing them in the attention layers consistently allows the model to outperform the original Transformer. We refer to this structure as Low-dimensional Projected Attention (LPA) , provide an explanation, and experimentally demonstrate its ability to reliably enhance both the efficiency and effectiveness of the model. We validate the effectiveness of the LPA model on two Transformer model configurations, assessing both pre-training and downstream task performance. With a particular focus on the scalability of LPA model, we observe that it remains effective even when the model parameters scale up to 3B. Furthermore, our study explores the effects of the hyperparameter on LPA, the necessity of integrating the low-dimensional module into every sublayer of the attention layer, and how to distribute any extra parameters effectively. The code of this work will be publicly available at https://github.com/TsinghuaC3I/LPA . 2 Related Work Low-rank Parameter-efficient Fine-tuning. Parameter-efficient fine-tuning optimize only a tiny portion of parameters while keeping the majority of the neural network frozen Houlsby et al. (2019) ; Li and Liang (2021) ; Lester et al. (2021) ; Hu et al. (2021) ; Zaken et al. (2021) ; Ding et al. (2023a) , saving significant time and computational costs and achieving performance comparable to full parameter fine-tuning on many tasks Ding et al. (2023b) . Low-rank adaptation (LoRA) is one of the most effective and influential parameter-efficient fine-tuning methods, having found widespread application Dettmers et al. (2023) . The LoRA method involves freezing the weights \mathbf{W}_{0} of the pre-trained model while training two low-rank decomposition matrices \mathbf{W}_{u} and \mathbf{W}_{d} , resulting in the output of the LoRA module being represented as \mathbf{z}\leftarrow\mathbf{W}_{0}\mathbf{x}+\mathbf{W}_{u}\mathbf{W}_{d}\mathbf{x} . We drew inspiration from LoRA and its improvement works, adapting them to the pre-training process to enhance effectiveness and efficiency of the model. Low-rank Pre-training for Neural Network. Some efforts have focused on making pre-training more efficient by reducing the number of trainable parameters Lin et al. (2020) ; Yuan et al. (2020) , and after finding that modules with low-dimension often yield poor results Bhojanapalli et al. (2020) , many works have concentrated on combining two low-rank matrices to reduce the parameter count while keeping the module dimensionality constant Schotthöfer et al. (2022) ; Idelbayev and Carreira-Perpinán (2020) ; Zhao et al. (2023) ; Thangarasa et al. (2023) . Current research has predominantly emphasized refining pre-training methods for CNN networks Sui et al. (2024) ; Jaderberg et al. (2014) or employing smaller language models Kamalakara et al. (2022) . However, some studies have found that low-rank pre-training can negatively impact model performance and training effectiveness, leading to the use of low-rank updates to train high-rank networks or the introduction of low-rank decomposition in gradient for optimization Lialin et al. (2023) ; Zhao et al. (2024) . Additionally, Liu et al. 2024 introduces low-rank latent states in the attention layer, successfully optimizing the KV cache. We discover that the unsatisfactory performance of the direct low-rank pre-training stems from the lack of precise parameter reduction placement. This insight guides our further exploration into the impact of low-dimensional modules and their applications at various locations within the model on both effectiveness and efficiency. 3 Low-dimensional Projected Attention We use a low-dimensional module for replacing the original weight matrix, and observe the varying effects of incorporating the low-dimensional structure in different modules. We provide an explanatory analysis of these findings and propose the Low-dimensional Projected Attention (LPA). Additionally, we examine the efficiency of this approach. 3.1 Low-dimensional Module Figure 1: An illustration of the Low-dimensional Projected Attention (LPA). The calculations in softmax function measure the relationships between input tokens. The low-dimensional module is constructed by sequentially connecting two low-dimensional matrices. Specifically, given a predetermined hyperparameter r , which is typically less than \frac{d_{\text{in}}\times d_{\text{out}}}{d_{\text{in}}+d_{\text{out}}} , the low-dimensional module comprises two matrices \mathbf{W}_{A}\in\mathbb{R}^{d_{\text{in}}\times r} and \mathbf{W}_{B}\in\mathbb{R}^{r\times d_{\text{out}}} , where d_{\text{in}} and d_{\text{out}} represent the input and output dimensions of the parameter matrix, respectively. The input data \mathbf{x}\in\mathbb{R}^{L\times d_{\text{in}}} passes through \mathbf{W}_{A} and \mathbf{W}_{B} sequentially, and the forward propagation

Read the complete text