PEFT-Factory : Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models
Robert Belanec, Ivan Srba, Maria Bielikova · 2025
Read PEFT-Factory : Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models by Robert Belanec, Ivan Srba, Maria Bielikova (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02764.txt.
Introduction
Robert Belanec, Ivan Srba, and Maria Bielikova unify the many small methods used to fine tune large language models. Parameter efficient fine tuning adapts huge models by training only a tiny added part. New variants appear constantly, and they are hard to replicate, deploy, or compare fairly.
PEFT-Factory gathers off the shelf and custom methods in one framework for autoregressive models. A shared home makes head to head tests and real deployments simpler. Practitioners with modest hardware gain most, since efficient tuning is their main route to models fitted for a language or task. One rack for all racks means a new trick can be tried on ten tools in a day, not in ten weeks of work.
Catalog record
| CREATOR | Robert Belanec, Ivan Srba, Maria Bielikova |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-02764
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PEFT-Factory : Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models Robert Belanec, Ivan Srba, Maria Bielikova Abstract Parameter-Efficient Fine-Tuning (PEFT) methods address the increasing size of Large Language Models (LLMs). Currently, many newly introduced PEFT methods are challenging to replicate, deploy, or compare with one another. To address this, we introduce PEFT-Factory , a unified framework for efficient fine-tuning LLMs using both off-the-shelf and custom PEFT methods. While its modular design supports extensibility, it natively provides a representative set of 19 PEFT methods, 27 classification and text generation datasets addressing 12 tasks, and both standard and PEFT-specific evaluation metrics. As a result, PEFT-Factory provides a ready-to-use, controlled, and stable environment, improving replicability and benchmarking of PEFT methods. PEFT-Factory is a downstream framework that originates from the popular LLaMA-Factory, and is publicly available at https://github.com/kinit-sk/PEFT-Factory . 1 Introduction Large Language Models (LLMs) ( Minaee et al. 2024 ; Radford et al. 2019 ; Dubey et al. 2024 ; Raffel et al. 2020 ) achieved remarkable results in many different Natural Language Processing (NLP) tasks mainly after the introduction of the transformer architecture ( Vaswani 2017 ) . However, for its great scaling capabilities ( Kaplan et al. 2020 ) , the size of the model in terms of trainable parameters is continuously increasing accordingly. The growing number of LLM parameters rendered fine-tuning computationally expensive, data-hungry, and hardly accessible for many researchers and practitioners. Parameter-Efficient Fine-Tuning (PEFT) methods ( Xu et al. 2023 ; Ding et al. 2023 ; Lialin et al. 2023 ; Han et al. 2024 ) aim to address these issues by training only a small percentage of the full model’s parameters while achieving performance comparable to that of full model fine-tuning. Such a decrease in trainable parameters can be achieved by adding new parameters ( Houlsby et al. 2019 ) , selecting specific parameters ( Ben Zaken et al. 2022 ) for training, or by reparameterizing the model with a smaller number of parameters ( Hu et al. 2022 ) . Due to their effectiveness, PEFT methods have gained popularity and become an attractive research area, with many new contributions being introduced each year ( Han et al. 2024 ) . However, the large number of newly introduced PEFT methods makes it harder to compare their contributions, resulting in only a few established PEFT methods (LoRA variants in most cases) being used in practice, while others, which may be more effective, remain largely unused. Moreover, many new PEFT methods lack a fully functional open-source implementation, and essential details on the experimental setup often prevent fellow researchers from replicating their results. Therefore, many researchers ( Asai et al. 2022 ; Shi and Lipani 2024 ; Tang et al. 2025 ) have to rely on reported performance metrics, while there is a risk of not replicating exactly the same experimental setup for their own methods (making a comparison potentially unfair), as well as it is not feasible to rerun the existing solutions on additional datasets/tasks. To tackle these accumulating problems, we introduce PEFT-Factory , an easy-to-use and modular framework for efficient fine-tuning and evaluation of LLMs using different PEFT methods . PEFT-Factory is based on the popular and open-source fine-tuning framework LLaMA-Factory ( Zheng et al. 2024b ) . It is built using PyTorch ( Paszke et al. 2019 ) and utilizes open-source Python modules for training LLMs, including Transformers ( Wolf et al. 2020 ) , PEFT ( Mangrulkar et al. 2022 ) , TRL ( von Werra et al. 2020 ) , and Adapters ( Poth et al. 2023 ) . Our main contributions are as follows: • PEFT-Factory provides a support for off-the-shelf methods from popular PEFT provider frameworks like HuggingFace PEFT ( Mangrulkar et al. 2022 ) or Adapters ( Poth et al. 2023 ) as well as dynamic loading of custom user-created PEFT methods . In contrast to the existing solutions, it provides so-far-missing support for soft prompt-based , adapter-based and selective PEFT methods; as well as for classification tasks. • PEFT-Factory natively provides a representative set of 19 PEFT methods , 27 classification and text generation datasets addressing 12 unique tasks , and standard as well as PEFT-specific evaluation metrics . This ready-to-use setup enables quick adoption and experimentation by researchers and practitioners, significantly improving the currently limited replicability and benchmarking of PEFT methods. • PEFT-Factory is designed with future extensibility in mind and provides a fully open-source codebase for anyone to use. It implements a standardized PEFT interface to enable modular addition of newly created PEFT methods. Similarly, it allows easy extension for additional datasets. Reparametrized Soft Prompt-Based Adapter-Based Selective Classification Datasets Classification Metrics Extensibility PEFT-Factory 8 5 4 2 ✔ ✔ datasets, models, PEFT methods LLaMA-Factory 7 0 0 0 ✘ ✘ datasets, models FastChat 3 0 0 0 ✘ ✘ datasets, models LitGPT 2 0 0 0 ✔ ✔ datasets, models LMFlow 3 0 0 0 ✘ ✘ datasets, models Axolotl 2 0 0 0 ✘ ✘ datasets, models Open-Instruct 3 0 0 0 ✘ ✘ ✘ H2O LLM Studio 3 0 0 0 ✔ ✔ datasets, models GPT4All 2 0 0 0 ✘ ✘ ✘ Table 1: Comparison of PEFT-Factory to popular fine-tuning frameworks. Only PEFT-Factory allows for out-of-the-box non-reparametrization efficient fine-tuning with the extensibility of additional and custom fine-tuning methods. Comparison at the level of individual PEFT methods can be found in Table 3 of Appendix B . 2 Related Work There has been a significant rise in the number of frameworks used for training of (not only) LLMs. LLaMA-Factory ( Zheng et al. 2024b ) is a recent addition to such frameworks and offers end-to-end and easy-to-use training of LLMs ranging across all of the stages (from pre-training to alignment via reinforcement learning). LLaMA-Factory also provides a graphical user interface called LLaMABoard, implemented in Gradio ( Abid et al. 2019 ) , which enhances the ease of use of LLaMA-Factory. Despite being a really popular and useful tool for LLM training, LLaMA-Factory still provides fine-tuning only with LoRA ( Hu et al. 2022 ) and its variants, namely QLoRA ( Dettmers et al. 2023 ) , DoRA ( Liu et al. 2024 ) , LoRA+ ( Hayou et al. 2024 ) , PiSSA ( Meng et al. 2024 ) , and GaLore ( Zhao et al. 2024b ) . With the recent update, LLaMA-Factory also allows Orthogonal Fine-Tuning (OFT) ( Qiu et al. 2023 ) , which utilizes the Cayley transformation ( Cayley 1846 ) to fine-tune only orthogonal vectors. Nevertheless, the selection of PEFT methods in LLaMA-Factory still remains limited. Lastly, LLaMA-Factory primarily focuses on text-generation problems and does not incorporate the possibility of casting text-generation problems as classification tasks. Our framework PEFT-Factory addresses both the limited number of available PEFT methods and the potential for fine-tuning LLMs for classification problems. There are also other LLM training frameworks that are less easy to run (compared to LLaMA-Factory) and have their specific benefits. FastChat ( Zheng et al. 2023 ) is a specialized framework for training LLMs for chat-completion. LitGPT ( AI 2023 ) and LMFlow ( Diao et al. 2023 ) are extensible and convenient general training frameworks that support various generative models and training methods. Axolotl