Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models
Xiaoshuang Ji, Zhendong Zhao (🖂), Xiaojun Chen, Xin Zhao, Zeyao Liu · 2025
Read Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models by Xiaoshuang Ji, Zhendong Zhao (🖂), Xiaojun Chen, Xin Zhao, Zeyao Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21119.txt.
Introduction
Xiaoshuang Ji and co-authors propose a progressive way to fine-tune transformer language models in a 2025 paper. Fine-tuning adapts pretrained models to downstream tasks, but updating every parameter grows costly as models enlarge. Parameter-efficient methods update only a small subset, yet they still update the same number of parameters throughout training.
That uniform treatment ignores the unequal contribution of different transformer blocks. ProgTuning instead grows or focuses the trainable set progressively as training proceeds. Spending the parameter budget where it matters most could match full fine-tuning quality at a fraction of the cost, which matters for teams without large compute.
Catalog record
| CREATOR | Xiaoshuang Ji, Zhendong Zhao (🖂), Xiaojun Chen, Xin Zhao, Zeyao Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-21119
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models Xiaoshuang Ji, Zhendong Zhao (🖂), Xiaojun Chen, Xin Zhao, Zeyao Liu Abstract Fine-tuning is a promising technique for leveraging Transformer-based language models in downstream tasks. As model sizes continue to grow, updating all model parameters becomes increasingly costly. Parameter-efficient fine-tuning methods effectively address this issue by selectively updating a small subset of parameters. However, fine-tuning and most existing parameter-efficient fine-tuning methods require updating the same number of parameters as the initial size, ignoring the unequal contribution across Transformer blocks and leading to extremely inefficient allocation of computing resources. In this paper, we propose Progtuning, the novel fine-tuning framework combined with progressive learning for Transformer-based language models. Specifically, Progtuning progressively reduces the number of updated transformer blocks based on the contribution. Remarkably, Progtuning optimizes resource allocation and reduces the number of updated parameters by approximately 25%, while still maintaining competitive performance. And it also exhibits high adaptability with parameter-efficient fine-tuning methods, demonstrating excellent performance across various adaptation scenarios. 1 Introduction Transformer-based Language Models [ 3 , 2 , 28 , 30 ] have achieved great success in the realm of Natural Language Processing (NLP). The models have been previously trained on general corpuses to learn general features, patterns, and representations. Notable examples of Transformer-based language models include GPT-3 [ 3 ] , T5 [ 22 ] , and LLaMA [ 21 ] . Fine-tuning is the predominant method to apply Transformer-based language models to down-stream tasks [ 31 ] . However, fine-tuning updates all parameters and stores a full copy for each task, which is gradually unacceptable as the size of models grows exponentially. For example, GPT-2 [ 1 ] has 1.5B parameters and GPT-3 [ 3 ] has 175B parameters just after one year. Fine-tuning requires extremely expensive calculation and storage resources. To address this issue, researchers have proposed various methods, including model pruning [ 32 , 33 ] , parameter quantization [ 34 , 35 ] and parameter-efficient fine-tuning [ 36 , 18 ] . Model pruning identifies and removes parameters or neurons that are less important, without degrading model performance. The primary objective of pruning is to minimize resource consumption and storage requirements, making it suitable for deployment on resource-constrained devices like mobile phones and IoT devices. Parameter quantization reduces the precision of the numbers used to represent a model’s parameters, for example, from floating-point precision (e.g., 32-bit floats) to lower precision (e.g., 8-bit integers). The most common methods are FP8/INT8 [ 34 ] and FP4/NF4/INT4 [ 35 ] . Parameter-efficient fine-tuning (PEFT) methods only update a small set of trainable parameters. The trainable parameters can be a set of newly introduced modules or a subset of parameters selected from the pretrained parameters [ 36 ] . PEFT achieves competitive performance compared with fine-tuning by updating barely 0.1%-5% task-specific parameters [ 18 ] . At present, parameter-efficient fine-tuning is the most promising method. Figure 1: The performance of Transformer blocks. We only update embedding layers and selected Transformer block in fine-tuning. The model we use is BERT_{BASE} and trained on SST-2 dataset. However, fine-tuning and most existing PEFT methods update all trainable parameters as the initial size during the whole training process, ignoring the unequal contribution to model performance across Transformer blocks. And this results in non-optimal allocation of computing resources. To illustrate this point, we provide a simple example in Fig. 1 . It can be concluded that (1) the performance of Transformer blocks is different, (2) high Transformer blocks achieve better performance than low blocks. The results motivate us to selectively allocate computing resources to high blocks that have more impact on model performance. Progtuning learning [ 10 ] is a widely utilized technique for stabilizing the training process in the field of computer vision. The key thought of progressive learning is to train models on simpler tasks (e.g., small images or models) and gradually extend to more complex tasks (e.g., large images or models). And it has been widely used in super-resolution [ 8 ] , facial attribute editing [ 9 ] and image synthesis [ 10 ] . In addition to stabilize the training process, computation operations are reduced inherently by progressive learning because the trained part of models is progressively changing. In this work, we propose Progtuning, the first fine-tuning framework that integrates progressive learning to gradually adjust the number of updated Transformer blocks during the training process, thereby allocating computing resource selectively. Before the training process starts, we divide the Transformer blocks into several parts based on the number of epochs. These parts are then organized into sequential stages, characterized by a progressive reduction in the number of Transformer blocks. Therefore, high Transformer blocks receive more updates than low blocks during fine-tuning. Progtuning demonstrates remarkable adaptability in combining with PEFT methods, such as Adapter tuning [ 5 ] , BitFit [ 13 ] and LoRA [ 7 ] . Our experiments show that Progtuning can reduce updated parameters effectively, concurrently achieving excellent model performance. And Progtuning seamlessly integrates with PEFT methods. Our contributions are summarized as follows. • We propose Progtuning, the first progressive fine-tuning framework. It gradually adjusts the amount of trainable Transformer blocks to allocate computing resources selectively. • Extensive experiments are conducted on various datasets and model architectures. Results show that Progtuning reduces updated parameters by about 25%. At the same time, the performance of models has been slightly improved. Progtuning also reduces training time greatly due to the reduction of updated parameters 1 1 1 Since the training time does not directly reflect the consumption of computing resources, we did not include it in the experiment. . • In addition, further experiments are conducted to explore the performance of combining Progtuning with PEFT methods. Experimental results show that the combination of Progtuning and PEFT methods works perfectly, which proves that Progtuning is of high adaptability with PEFT methods. 2 Related Work 2.1 Transformer-based Language Model A Transformer-based language model usually consists of the embedding layers E , several Transformers blocks and the classification header H , as illustrated in Fig. 2 . There are stacks of L Transformer blocks. The embedding layers E transform the input token lists and positional encoding into embedding vectors and then, Transformer blocks extract semantic information individually. Transformer block is the basic unit of pretrained language models. A single Transformer block mainly consists of two parts: Self-Attention layer and Feed Forward Neural Network (FFN). Both Self-Attention layer and FFN have residual connections [ 37 ] and Layer Normalization [ 38 ] . The input first passes through the Self-Attention layer to get the attention score and then is passed to the next Transformer block after being activated by FFN. At last, the classification header H predicts the probability distribution of the next token. Figure 2: The structure of a Transformer-based language model. 2.2 Fine-tuning Fine-tuning is the predominant model training technique used in the realm of NLP [ 16 , 17 ] . Pretrained models are trained on specific datasets to acquire domain-specific knowledge of downstream tasks. Fine-tuning modifies all parameters and saves a full copy of pretrained parameters for each task. Since the model has been pretrained on the general corpuses, the number of