Loss-Minimizing Model Compression via Joint Factorization Optimization
Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangming Liu, Jiake Tian · 2024
Read Loss-Minimizing Model Compression via Joint Factorization Optimization by Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangming Liu, Jiake Tian (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06867.txt.
Introduction
Boyang Zhang and co-authors present a way to shrink deep learning models by factoring their weight tables, without the usual small loss of accuracy, in a 2024 paper. Splitting a weight table into two thinner ones cuts size, but tuning the split alone leaves a gap from the original model's skill. The team traces that gap to the split being tuned apart from the model's own training goal.
Their fix tunes both at once under one joint goal. First they work out how far split errors may stray before model skill moves. They then offer two routines: one that maximizes accuracy while compressing, and one that minimizes size while holding skill steady. Neither needs retraining. In one test, a model shrunk by 70 percent beat the original it came from, across vision and language tasks.
Catalog record
| CREATOR | Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangming Liu, Jiake Tian |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06867
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Loss-Minimizing Model Compression via Joint Factorization Optimization Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangming Liu, Jiake Tian Abstract Low-rank factorization is a popular model compression technique that minimizes the error δ \delta between approximated and original weight matrices. Despite achieving performances close to the original models when δ \delta is optimized, a performance discrepancy remains due to the separate optimization processes for low-rank factorization and model performance, resulting in unavoidable losses. We address this issue by introducing a novel joint optimization strategy for lossless low-rank weight factorization, which, for the first time, enhances the model’s performance beyond the original. Our approach begins with a theoretical analysis of the relationship between low-rank factorization and model optimization objectives, establishing a precise perturbation range for matrix factorization errors on model performance. This challenge is then reformulated as a numerical rank deficiency problem with inequality constraints and develop a joint objective that simultaneously addresses factorization error and model performance. Based on the above analysis, we propose two optimization algorithms: a lossless optimization algorithm that maximizes model accuracy while ensuring compression, and a compact optimization algorithm that minimizes model size while preserving performance. These algorithms do not require fine-tuning and can directly compress numerous deep models to achieve lossless results. Our methods demonstrate robust efficacy across various vision and language tasks. For example, the compressed model reduced by 70% on ResNext50 outperforms the original. Our code will be made public. 1 Introduction Deep neural networks have excellent performance in language and vision tasks. A common problem is the significant increase in the number of parameters, which brings challenges to deployment and inference. Matrix factorization is a common and promising compression method in the community. The idea of matrix factorization is to decompose the weight matrix into two or more smaller matrices and use these two small matrices in actual storage and calculation. Compared with other compression methods such as pruning and quantization, matrix factorization has solid mathematical theoretical support and can fully preserve the intrinsic structure and potential information of the data. Traditional matrix factorization methods include factorization, Singular Value Decomposition (SVD) [ 7 ] , and eigenvalue decomposition. When matrix factorization is applied to neural networks, these methods are extended to CANDECOMP/PARAFAC (CP) [ 17 ] , Tucker decomposition of kernel tensors. These methods are friendly to linear layers and are often used to compress neural networks. Figure 1 : The left subfigure shows the process of factorization, \delta is the noise error introduced by the factorization. The right subfigure shows the Loss comparison between our algorithm and existing factorization algorithms. Our algorithm factorizes models losslessly. L is the model loss. There are two main types of existing post-training low-rank compression schemes. The first scheme is to directly decompose the existing model weights, such as SVD, CANDECOMP/PARAFAC, Tensor-Train (TT), Tucker decomposition, etc., but this type of scheme will lead to increased loss and poor model performance, such as a 5-10 times increase in loss. To improve this scheme, the second scheme is proposed, that is, after direct weight factorization, different strategies are used to fine-tune the model on the entire dataset. Finetuning is to decompose the model and retrain it using the entire data set. Several studies have contributed to this second scheme. For instance, Yu et al. [ 23 ] considered weight structure information and combined low-rank weight matrix and feature map reconstruction to reduce fully connected layer parameters. Xu et al. [ 20 ] integrated low-rank approximation and regularization into training with less performance loss. Yang et al. [ 21 ] proposed SVD training, decomposing each layer into a full-rank form, and then directly training the decomposed weights. Hsu et al. [ 5 ] used Fisher information to measure the importance of model parameters and weight them into singular value decomposition. Yu et al. [ 22 ] adaptively determined the structure of the compression model using a small number of training samples, thereby compressing the output features of each linear layer. Kim et al. [ 9 ] recovered the lost precision through Tucker decomposition on the nuclear tensor and fine-tuning. Zhang et al. [ 25 ] used multiple low-rank matrices to approximate a complete gated recurrent unit weight matrix and retrain. Although this scheme can improve the model performance when compressing the model, it requires a large amount of fine-tuning data and the fine-tuning process can be time-consuming. It is also not possible to compress the model losslessly, i.e., whether fine-tuned or not, the loss value of the compressed model is always greater than the original model value. This is because the optimization processes of matrix factorization and model learning are different and separate. These methods follow the “Factorization + Finetuning” paradigm. The optimization goal of factorization is to minimize the approximate weight error \delta , while the optimization goal of model learning is to minimize the loss function L . According to the “Factorization + Finetuning” paradigm, the factorization and fine-tuning processes are completed in two stages. Therefore, the approximate error and model loss are separate and have no direct relationship, that is, when the approximate error is minimized, the model loss is not necessarily the lowest, as shown in Figure 2 . Unlike previous work, our method considers the joint low-rank factorization and model learning objectives, does not require training and fine-tuning, and establishes a joint optimization process, thereby compressing the model losslessly for the first time, as shown in Figure 1 . Figure 2 : Weight error and loss values of different models, where Rank1 > Rank2 > Rank3 > Rank4 . When the weight error \delta is the lowest, the model loss L does not reach the minimum value. Based on the above analysis, we propose a lossless joint low-rank factorization strategy without fine-tuning. Our strategy first establishes the connection between the optimization objectives of low-rank factorization and model learning in theory. The theoretical limitations in practice are then converted into inequality constraints for the optimization problem. Specifically, we determine the perturbation range of the loss caused by the factorization error in each layer of the model. Taking this perturbation range as the calculus neighborhood, the connection between the low-rank factorization and the optimization of the original model is mathematically established. Then by imposing error constraints, the factorization optimization problem is converted into a numerical rank-defect optimization problem under inequality constraints, and a new objective related to model performance is proposed. For different requirements, we design two algorithms to solve this problem, lossless optimization and compact matrix optimization algorithms under numerical rank-defect. The lossless optimization algorithm aims to find the lowest loss model for the current layer under model compression. The compact matrix optimization algorithm aims to find the most compact model for the current layer under lossless conditions. It is worth noting that both algorithms do not require fine-tuning and can obtain lossless layers or models. In summary, the contributions of this paper are as follows: 1) We mathematically establish the connection between factorization and model learning objectives, and propose a lossless joint low-rank factorization strategy that does not require fine-tuning. 2) We convert the traditional factorization optimization problem into a numerical rank-defect optimization problem under inequality constraints and propose two algorithms for different requirements. 3) Extensive