Towards Efficient Post-Training via Fourier-Driven Adapter Architectures

Donggyun Bae Jongil Park, Konkuk University, Corresponding author: [email protected] · 2025

Read Towards Efficient Post-Training via Fourier-Driven Adapter Architectures by Donggyun Bae Jongil Park, Konkuk University, Corresponding author: [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22378.txt.

Introduction

Donggyun Bae and Jongil Park propose Fourier-Activated Adapters, a way to fine-tune large language models by training only small added modules. Their adapters inject random Fourier features into the model's intermediate representations, splitting them into low- and high-frequency components. The design lets adaptation emphasize whichever frequency bands carry useful information for a task while the frozen backbone keeps its general capacity.

Across GLUE, E2E NLG, and instruction-tuning benchmarks, the method matched or beat current parameter-efficient fine-tuning approaches while keeping computation and memory overhead low. Ablation studies confirmed that both the frequency-aware activation and the adaptive weighting methods contribute to the gains. The authors present the system as a robust, efficient option for post-training large models.

Catalog record

CREATORDonggyun Bae Jongil Park, Konkuk University, Corresponding author: [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22378

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Efficient Post-Training via Fourier-Driven Adapter Architectures Donggyun Bae Jongil Park, Konkuk University, Corresponding author: [email protected] Abstract We propose a novel framework, named Fourier-Activated Adapter (FAA) for parameter-efficient fine-tuning of larg-secale pre-trained language models. By integrating random Fourier features into the adapter module, our FAA decomposes input representations into high- and low-frequency components and employs a dynamic, frequency-aware activation mechanism to selectively emphasize crucial semantic signals. Our FAA improves the performance of the fine-tuned model and enhances the model’s perception of multi-frequency semantic information. Experiments on GLUE, E2E NLG, and instruction tuning benchmarks demonstrate competitive or superior results of our FAA. Besides, ablation studies confirm the importance of frequency-aware activation and adaptive weighting. This demonstrates that our FAA is an effective and robust solution for enhancing the performance of large language models while maintaining computational efficiency. Figure 1: Comparison of traditional adapters and Fourier-activated adapters (FAA). The top shows the traditional adapter, which consists of a feed-forward down-projection, a nonlinearity (e.g., a GELU function), and a feed-forward up-projection. The bottom shows the Fourier-activated adapter (FAA). The fixed activation function is replaced by RFF+GELU, where RFF first controls the consistency of the generated frequency features in the numerical scale through the normalization factor \sqrt{\frac{2}{D_{\text{rff}}}} , then controls the frequency distribution through the \beta coefficient, and then combines with GELU through learnable coefficients a and b to achieve a frequency-aware Fourier activation strategy. Note that the adapter layers are interspersed between the attention layer and the feed-forward network. 1 Introduction Large Language Models (LLMs) have become a fundamental technology in NLP, demonstrating exceptional language comprehension and generation capabilities 2 ; 62 . With the rise of large-scale pre-trained models, LLMs have achieved remarkable progress in machine translation 65 ; 59 , question answering 1 ; 50 ; 49 , and text generation 24 ; 60 . However, they still face challenges in cross-domain generalization, out-of-distribution robustness 47 ; 51 ; 61 , and low-resource scenarios. When dealing with complex tasks or scarce data, fine-tuning large models directly not only requires extensive computational resources but also leads to suboptimal generalization, making it difficult to handle domain-specific terminology, intricate syntax, and sudden semantic shifts. To address these issues, Parameter-Efficient Fine-Tuning (PEFT) techniques 7 ; 45 have been introduced. By freezing most of the base model’s parameters and only introducing lightweight adapter modules 18 for fine-tuning, PEFT significantly reduces computational and storage costs while improving deployment efficiency. As an effective solution, PEFT enables LLMs to maintain high performance across various downstream tasks while minimizing computational demands. However, existing PEFT methods still struggle to capture high-frequency semantic information and handle complex tasks, limiting their effectiveness in cross-domain generalization and low-resource scenarios. Current PEFT approaches face two major challenges. First, traditional adapter architectures are mostly designed with fixed activation functions 17 ; 57 , making them rigid in responding to frequency variations in input data. This prevents them from dynamically adapting to high-frequency features in complex tasks, thereby affecting performance in specialized domains, intricate syntactic structures, and scenarios with rapid semantic changes. Second, while low-rank optimization methods (e.g., LoRA 19 ; 55 ; 56 ; 53 ) achieve compression by reducing parameter count, they fail to fully leverage frequency-domain structures. As a result, these models struggle to capture fine-grained, high-frequency semantic features, making it difficult to maintain representation precision. Through an in-depth analysis of LLM behavior, we observe clear spectral sparsity in text processing 41 ; 54 ; 52 , where core semantic information is often concentrated in a few key frequency bands, while most low-frequency components contribute little to meaningful representation. Previous works ( 42 ; 48 ; 46 , 23 ; 11 ; 10 ) have attempted to incorporate frequency-based enhancements into language models, yielding promising results. Meanwhile, the introduction of Kolmogorov-Arnold Networks (KAN) 30 has provided new insights into learning adaptive activation functions. The subsequent development of Kolmogorov-Arnold Fourier Networks (KAF) 58 ; 9 ; 3 builds on KAN by integrating Fourier-based enhancements to improve spectral analysis capabilities. Motivated by these findings, we propose a new approach that utilizes Random Fourier Features (RFF) 37 ; 25 ; 26 to transform input signals into the frequency domain, reconfiguring the traditional adapter processing pipeline. Our goal is to design a frequency-domain processing module that introduces only a minimal parameter overhead to the base model while significantly enhancing generalization in cross-domain and low-resource scenarios. The proposed Fourier-Activated Adapter Framework consists of three key components. First, inspired by KA activation strategies, we construct a dynamic frequency-aware activation mechanism that allows the adapter to adjust its response to critical information dynamically across different tasks. Second, we introduce a randomized Fourier transform to decompose input signals in the frequency domain, leveraging spectral sparsity to effectively map high-frequency features into a lower-dimensional space. Finally, we develop a lightweight training strategy that incorporates sparsity constraints, reducing the parameter overhead introduced by Fourier transform while keeping the base model intact. Experimental results demonstrate that our FAA Framework achieves significant improvements in cross-domain generalization and low-resource tasks. The proposed framework not only effectively reduces the number of parameters required for fine-tuning but also achieves performance comparable to or even surpassing traditional full fine-tuning methods across multiple tasks. By leveraging this novel technique, we provide an efficient and practical solution for deploying LLMs in real-world applications, particularly in resource-constrained environments and multi-task processing scenarios. 2 Related Work 2.1 Parameter-Efficient Fine-Tuning Techniques Parameter-Efficient Fine-Tuning (PEFT) methods have gained widespread application in the adaptive adjustment of large-scale pre-trained language models in recent years. Traditional full-parameter fine-tuning methods 28 ; 31 ; 14 ; 4 require updating a large number of model parameters when dealing with specific tasks, leading to high computational and storage costs. To address this issue, researchers have proposed various PEFT methods, such as Adapters 18 and LoRA 19 . Adapters insert lightweight adapter modules between the layers of the model, fine-tuning only these new parameters, thereby significantly reducing the number of parameters required for fine-tuning. LoRA further reduces the scale of parameter updates through low-rank matrix decomposition. These methods improve the efficiency and flexibility of fine-tuning while maintaining model performance. 2.2 Frequency Domain Enhancement and Fourier Transform Frequency domain analysis, successful in computer vision 33 ; 44 , is gaining traction in NLP 42 . By transforming text signals into the frequency domain, these methods better capture high and low-frequency features, improving pattern understanding. Recent work 15 ; 20 has integrated Fourier transforms into language models, enhancing multi-frequency semantic modeling 22 and showing benefits in cross-domain and low-resource scenarios. Studies 13 ; 39 show that key semantic information concentrates in specific frequency bands, with methods like FourierFT 12 decomposing inputs to better capture multi-frequency components. However, current approaches have not fully leveraged frequency domain structures for semantic representation, making the optimization of these techniques an important research direction. 2.3 Kolmogorov-Arnold Networks (KAN) and Fourier Activation Kolmogorov-Arnold Networks (KAN) 30 introduced a new activation mechanism that improves model response through adaptive learning. This evolution led to Kolmogorov-Arnold Fourier Networks (KAF) 58 , which combines Fourier transforms with a frequency-aware activation mechanism, allowing dynamic adjustment to different frequency information and better capture of high-frequency details. Building on these advances, we propose a Fourier-Activated Adapter framework (FAA)

Read the complete text