TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, Dacheng Tao · 2025
Read TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs by Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, Dacheng Tao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19057.txt.
Introduction
Yan Sun and co-authors published a computing paper in 2025 on fine-tuning large language models without gradient calculations. Zeroth-order methods tune models cheaply, and recent versions exploit low-rank structure in each step's gradient. The team notes all gradients across training share nearly one subspace, a fact earlier work ignored. Their method, TeZO, captures low-rank structure across both model and time.
TeZO stores step perturbations as a three-way array and splits it to extract low-rank matrices, cutting training cost. An Adam variant uses about a third of a rival's memory. Theory and broad tests back the gains, matching top results at lower time and memory cost.
Catalog record
| CREATOR | Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, Dacheng Tao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19057
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, Dacheng Tao Abstract Zeroth-order optimization (ZO) has demonstrated remarkable promise in efficient fine-tuning tasks for Large Language Models (LLMs). In particular, recent advances incorporate the low-rankness of gradients, introducing low-rank ZO estimators to further reduce GPU memory consumption. However, most existing works focus solely on the low-rankness of each individual gradient, overlooking a broader property shared by all gradients throughout the training, i.e., all gradients approximately reside within a similar subspace. In this paper, we consider two factors together and propose a novel low-rank ZO estimator, TeZO , which captures the low-rankness across both the model and temporal dimension. Specifically, we represent ZO perturbations along the temporal dimension as a 3D tensor and employ Canonical Polyadic Decomposition (CPD) to extract each low-rank 2D matrix, significantly reducing the training cost. TeZO can also be easily extended to the Adam variant while consuming less memory than MeZO-SGD , and requiring about only 35 % 35\% memory of MeZO-Adam . Both comprehensive theoretical analysis and extensive experimental research have validated its efficiency, achieving SOTA-comparable results with lower overhead of time and memory. 1 Introduction As the model size progresses at an extraordinary rate ( Zhang et al., 2022 ; Touvron et al., 2023 ; Achiam et al., 2023 ) , memory and computational resources have become the primary bottleneck limiting development. In response to this challenge, ZO has opened up new possibilities for efficient training ( Shen et al., 2023 ) . Adopting gradient-free updates with a small amount of high-quality data perfectly unlocks the knowledge of the entire domain, offering significant potential for several practical applications. Since Spall (1992) introduced ZO as a promising alternative to FO in the training process, it has been widely applied in gradient-computation-challenged scenarios ( Wang et al., 2018 ; Liu et al., 2020 ) and in black-box optimization ( Chen et al., 2017 ; Tu et al., 2019 ) . Recent studies have also highlighted the great potential of ZO in fine-tuning LLMs. Malladi et al. (2023) propose the MeZO method which adopts classical ZO-SGD ( Ghadimi & Lan, 2013 ) for fine-tuning. Furthermore, it reduces memory costs by only preserving random seeds instead of variables. Compared to FO, it can achieve comparable performance while requiring approximately 10% of memory in practice, greatly improving memory efficiency. (a) Low-rankness on model dimension. (b) Low-rankness on temporal dimension. (c) Memory usage on training OPT-13B. Figure 1: (a) and (b) are validation of the low-rankness of gradients. We fine-tune OPT-1.3B on SST-2 and calculate top-100 singular values of gradients of layers.9.self_attn.out_proj.weight . We then concatenate these singular value vectors and display them as a heat-map in (a). Then we concatenate the normalized gradient of each layer over a total of T iterations into a matrix with the size of d_{l}\times T , calculate the top-100 singular values corresponding to layers and display them as a heat-map in (b). In (c), we record the GPU memory usage of MeZO , our TeZO , and corresponding variants on training OPT-13B model. We also provide more interesting experiments on the low-rankness and studies of subspace of gradients on LLaMA-7B in Appendix A.1 . Although ZO has made significant progress, it still faces two main challenges, i.e., i) lack of detailed characterization of gradients; ii) the costs of optimization states to generate random variables significantly increase as d grows. This also highlights the bottleneck of ZO methods in LLM tasks. Recent advances learned the strong low-rank nature of gradients in LLMs ( Wang et al., 2023 ; Jaiswal et al., 2024 ) , making low-rank representations in ZO methods as an ingenious solution to the aforementioned issues. With barely compromising performance, low-rank ZO methods effectively reduce the required memory for ZO estimations from \mathcal{O}(d) to \mathcal{O}(\sqrt{d}r) at most, where r is the rankness constant ( Chen et al., 2024 ; Yu et al., 2024 ) . This implementation further endows the ZO method with superior value in the tasks of fine-tuning LLMs. Nonetheless, this technique still exhibits substantial potential for further developments. Our Motivations. Existing methods only consider each individual gradient to be low-rank. From the entire training perspective, the total cost achieves \mathcal{O}(\sqrt{d}T) after T iterations. Although techniques like lazy updates reduce the frequency of generating factor vectors, training costs still expand linearly with \sqrt{d} . It indicates that merely considering the low-rank form of each individual gradient is insufficient. This inspires our contemplation: could the low-rankness be incorporated along the temporal dimension as well? To investigate an efficient approach to addressing this question, in this paper, we comprehensively study the characteristics of the gradients in LLMs. As shown in Figure 1 .(a) and (b) , in the tasks of fine-tuning LLMs, the gradients exhibit the following two properties simultaneously: i) the individual gradient at each iteration is approximately low-rank; ii) all gradients along T iterations lie almost within a similar subspace. Obviously, combining properties i) and ii) can lead to higher efficiency. Inspired by this, we propose the TeZO estimator to empower the low-rankness on the temporal dimension. Specifically, we estimate the ZO perturbations as a 3D tensor with the size of m\times n\times T . By adopting the Canonical Polyadic Decomposition (CPD) ( Hitchcock, 1927 ) , the 3D tensor can be estimated by the sum of r rank-1 tensors where r approximates its rank. The joint low-rankness significantly reduces the cost of factor vectors during computation. At each iteration t , we only need to generate temporal factor vector to extract a 2D matrix, further lowering the costs from \mathcal{O}(\sqrt{d}T) to \mathcal{O}(\sqrt{d}+T) . We also introduce an auxiliary technique to dynamically select the rank r_{l} for each layer. In addition, we propose both memory-efficient momentum-based and adaptivity-based variants, i.e. TeZO-m and TeZO-Adam , which demonstrate the high scalability of the TeZO method. As shown in Figure 1 .(c) , TeZO-Adam consumes less memory than MeZO-SGD , and requiring about only 35\% memory of MeZO-Adam . Both comprehensive theoretical analysis and extensive experimental research are conducted to validate its efficiency. TeZO-Adam achieves performance superior to MeZO-Adam and other ZO optimizers, while requiring only the memory overhead of general ZO-SGD . We summarize our contributions as follows: • By jointly considering low-rankness of gradients and their similarity on LLMs, we propose a novel low-rank ZO estimator, TeZO , which constructs the ZO perturbations via CPD to reduce the training overhead. • We introduce an auxiliary technique to dynamically select the rank for each layer and extend TeZO to two memory-efficient variants, TeZO-m and TeZO-Adam . Both require lower memory consumption and time cost than MeZO-SGD and significantly lower than the costs of corresponding variants in MeZO . • We prove that TeZO is an unbiased estimator of FO gradient, maintaining the comparable variance and convergence rate as existing ZO methods with less overhead. Extensive experiments are conducted to validate its fine-tuning efficiency on LLMs. 2 Related Work Zero-Order Optimization. Since Spall (1992) proposed the ZO method, it has been extensively studied and practically incorporated in various domains ( Chen et al., 2017 ; Tu et al., 2019