FPS: Feedforward-based Parameter Selection For Efficient Fine-Tuning
Kenneth Yang Wen-Li Wei Jen-Chun Lin Academia Sinica Academia Sinica Academia Sinica National Taiwan University [email protected] [email protected] [email protected] · 2025
Read FPS: Feedforward-based Parameter Selection For Efficient Fine-Tuning by Kenneth Yang Wen-Li Wei Jen-Chun Lin Academia Sinica Academia Sinica Academia Sinica National Taiwan University [email protected] [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27359.txt.
Introduction
Kenneth Yang, Wen-Li Wei, and Jen-Chun Lin speed up a cheap way to tune big AI models in this 2025 paper. The cheap way tunes only a chosen subset of the model's weights. Picking the subset by gradients, the usual slope signals, costs a full backward pass and lots of memory. The pick also shifts with random batch luck.
Their method, GD-FPS, makes the pick with forward passes only, so no gradients at all. It scales each weight by how much its activity grew since pretraining, measured against a fixed anchor. Across twenty-six vision tasks, the picks matched or beat rival methods. Next to the gradient method, it used about eighteen times less memory and ran over twice as fast at the picking stage. The pick is also fixed and repeatable.
Catalog record
| CREATOR | Kenneth Yang Wen-Li Wei Jen-Chun Lin Academia Sinica Academia Sinica Academia Sinica National Taiwan University [email protected] [email protected] [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-27359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FPS: Feedforward-based Parameter Selection For Efficient Fine-Tuning Kenneth Yang Wen-Li Wei Jen-Chun Lin Academia Sinica Academia Sinica Academia Sinica National Taiwan University [email protected] [email protected] [email protected] Abstract Parameter-Efficient Fine-Tuning (PEFT) has emerged as a key strategy for adapting large-scale pre-trained models to downstream tasks, but existing approaches face notable limitations. Addition-based methods, such as Adapters [ 1 ] , introduce inference latency and engineering complexity, while selection-based methods like Gradient-based Parameter Selection (GPS) [ 2 ] require a full backward pass, which results in the same peak memory usage as full fine-tuning. To address this dilemma, we propose Feedforward-based Parameter Selection (FPS), a gradient-free method that identifies an optimal parameter subset in a single forward pass. FPS ranks parameters by the product of their magnitudes and corresponding input activations, leveraging both pre-trained knowledge and downstream data. Evaluated on 24 24 visual tasks from FGVC and VTAB-1k, FPS achieves performance comparable to state-of-the-art methods while reducing peak memory usage by nearly 9 × 9\times and accelerating parameter selection by about 2 × 2\times , offering a genuinely memory-efficient and practical solution for fine-tuning large-scale pre-trained models. 1 Introduction Large-scale pre-trained models have become a cornerstone of foundation model research. These models, developed by training on vast and varied datasets, demonstrate impressive general capabilities across a wide range of tasks. To unlock their full potential and achieve state-of-the-art results for each downstream task, a crucial technique is fine-tuning. This process adapts these general-purpose models to specific tasks or domains, enabling specialization such as fine-tuning BERT [ 3 ] for sentiment analysis on movie reviews [ 4 ] or adapting a Vision Transformer (ViT) [ 5 ] to detect brain tumors from MRI scans [ 6 ] , among many other downstream applications [ 7 , 8 , 9 ] . However, the naive approach of fine-tuning all model parameters—commonly referred to as full fine-tuning—is often impractical due to two primary challenges. First, it poses a highly complex optimization challenge, as billions of parameters must be updated using the relatively small datasets typical of downstream tasks, often resulting in suboptimal performance. Second, full fine-tuning is a computationally and resource-intensive endeavor due to its massive memory usage. The associated costs are substantial, including the storage of the model parameters themselves, their gradients during backpropagation, optimizer states, and crucially, the intermediate activations required for gradient computation. This renders full fine-tuning computationally and memory prohibitive, and ultimately impractical for many applications. To mitigate the challenges of full fine-tuning, Parameter-Efficient Fine-Tuning (PEFT) has emerged as a leading approach. PEFT methods adapt models for downstream tasks by optimizing only a minimal set of parameters, often achieving performance that is comparable or even superior to full fine-tuning while incurring only a fraction of the computational cost. Many prominent PEFT techniques are addition-based, meaning they introduce new, trainable components into the existing model. For example, Adapters [ 1 ] insert small, learnable neural modules between the model’s layers. However, these methods generally face two significant issues. First is the additional overhead they introduce, which can increase the model’s storage size and inference latency. The second, and more critical, challenge is the engineering complexity, which arises from the considerable design difficulty of determining the optimal placement of these new modules [ 10 , 2 ] . This decision often relies on task-specific heuristics; for instance, tasks requiring high-level semantic refinement might benefit from inserting Adapters in deeper layers, whereas tasks focused on low-level features may be better served by placing them in shallower layers. Such reliance on expert tuning and model-specific knowledge undermines their potential as a simple “plug-and-play” solution. An alternative category, selection-based PEFT, has recently been proposed to address model overhead and engineering complexity. Our method falls into this group , with Gradient-based Parameter Selection (GPS) [ 2 ] being the closest prior work. The GPS process involves two stages: parameter selection followed by fine-tuning. To select parameters, it must first unfreeze the entire model to compute gradients for all parameters on the downstream task. Once the parameters with the highest gradient magnitudes are identified, only that subset is fine-tuned. This approach successfully eliminates additional parameters and the need for complex engineering design. However, the requirement to compute gradients for the full model during the selection stage contradicts the primary motivation of PEFT. It suffers from the same peak memory usage as full fine-tuning, which creates a dilemma. To resolve this, we propose the Feedforward-based Parameter Selection (FPS) method, a gradient-free approach that provides a genuinely memory-efficient solution. Concretely, the bottleneck of GPS lies in its selection stage, which requires a complete and computationally expensive forward–backward pass. This memory overhead arises from computing and storing gradients for the entire model during backpropagation. This raises a critical question: can we bypass this issue with a gradient-free approach? Why not determine parameter importance on-the-fly during the forward pass? This is the core idea behind our method. We propose selecting the optimal parameter subset based on the product of parameters and their corresponding input activations—an approach that jointly accounts for downstream data properties and pre-trained model weights. Since this criterion depends only on activation values, it can be computed in a single forward pass, yielding substantial gains in both memory efficiency and selection speed. We evaluate our approach on the FGVC and VTAB-1k benchmarks, covering a total of 24 visual tasks. Compared to GPS, our method attains comparable performance while reducing peak memory usage by nearly 9\times and accelerating the parameter selection stage by about 2\times , thereby providing a more resource-efficient and practical solution. 2 Related Works Following prior work [ 2 ] , we categorize PEFT methods into two main paradigms: addition-based methods , which augment the model with new trainable components, and selection-based methods , which identify and update a subset of the model’s original parameters. 2.1 Addition-based Methods Addition-based methods augment the model with new trainable components but typically incur two major drawbacks: increased inference overhead and substantial engineering complexity. The first drawback, increased inference overhead, arises in multiple forms. Adapter-based methods and their variants [ 1 , 11 , 12 , 13 , 14 ] insert additional modules directly into the model architecture, thereby enlarging both model size and computational graph, which results in slower inference. While Low-Rank Adaptation (LoRA) [ 15 ] mitigates this latency by merging low-rank matrices back into the original weights, it fails to address the second—and more critical—design challenge. The second drawback lies in the substantial engineering complexity, which limits these methods from being truly model- or task-agnostic. For approaches such as Adapters and LoRA, determining the optimal placement, dimensionality, and architecture of the new components presents a considerable design challenge that often depends on task-specific heuristics. Similarly, prompt-based methods, exemplified by Visual Prompt Tuning (VPT) [ 16 , 17 , 18 , 19 ] , inject learnable context tokens into the input sequence, introducing their own design challenges while also incurring inference overhead by lengthening the input. This becomes problematic for models not designed to accommodate variable input lengths, thereby undermining their claim of universally applicable “plug-and-play” solutions. 2.2 Selection-based Methods Selection-based methods, to which our proposed approach belongs, avoid introducing additional parameters. Early variants typically relied on simple heuristics, such as