Compressing Sine-Activated Low-Rank Adapters through Post-Training Quantization

Cameron GordonAustralian Institute for Machine LearningUniversity of Adelaide, Yiping JiAustralian Institute for Machine LearningUniversity of AdelaideDATA61, CSIRO, Hemanth SaratchandranAustralian Institute for Machine LearningUniversity of AdelaidePaul AlbertAustralian Institute for Machine Learni · 2025

Read Compressing Sine-Activated Low-Rank Adapters through Post-Training Quantization by Cameron GordonAustralian Institute for Machine LearningUniversity of Adelaide, Yiping JiAustralian Institute for Machine LearningUniversity of AdelaideDATA61, CSIRO, Hemanth SaratchandranAustralian Institute for Machine LearningUniversity of AdelaidePaul AlbertAustralian Institute for Machine Learni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21895.txt.

Introduction

Cameron Gordon and co-authors compress model updates for sharing between devices in a 2025 paper. In delta compression, parties share one base model and send only the changes, which must be tiny. Popular update methods use low-rank adapters, but those have a limited range that aggressive squeezing exposes badly.

The authors build on a sine-activated form of low-rank update that widens that range. They then quantize the trained update hard, cutting its precision to shrink size further, while holding quality. The pairing targets phones and edge devices that must receive fresh model behavior over thin links. Their results chart how small an update can get before the model's skill starts to slip.

Catalog record

CREATORCameron GordonAustralian Institute for Machine LearningUniversity of Adelaide, Yiping JiAustralian Institute for Machine LearningUniversity of AdelaideDATA61, CSIRO, Hemanth SaratchandranAustralian Institute for Machine LearningUniversity of AdelaidePaul AlbertAustralian Institute for Machine Learni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21895

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Compressing Sine-Activated Low-Rank Adapters through Post-Training Quantization Cameron GordonAustralian Institute for Machine LearningUniversity of Adelaide, Yiping JiAustralian Institute for Machine LearningUniversity of AdelaideDATA61, CSIRO, Hemanth SaratchandranAustralian Institute for Machine LearningUniversity of AdelaidePaul AlbertAustralian Institute for Machine LearningUniversity of AdelaideSimon LuceyAustralian Institute for Machine LearningUniversity of Adelaide Abstract Low-Rank Adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning, offering substantial reductions in trainable parameters by modeling updates as the product of two low-rank matrices. While effective, the low-rank constraint inherently limits representational capacity, often resulting in reduced performance compared to full-rank fine-tuning. Recent work by Ji et al. 2025 has addressed this limitation by applying a fixed-frequency sinusoidal transformation to low-rank adapters, increasing their stable rank without introducing additional parameters. This raises a crucial question: can the same sine-activated technique be successfully applied within the context of Post-Training Quantization to retain benefits even after model compression? In this paper, we investigate this question by extending the sinusoidal transformation framework to quantized LoRA adapters. We develop a theoretical analysis showing that the stable rank of a quantized adapter is tightly linked to that of its full-precision counterpart, motivating the use of such rank-enhancing functions even under quantization. Our results demonstrate that the expressivity gains from a sinusoidal non-linearity persist after quantization, yielding highly compressed adapters with negligible loss in performance. We validate our approach across a range of fine-tuning tasks for language, vision and text-to-image generation achieving significant memory savings while maintaining competitive accuracy. 1 Introduction Parameter-efficient fine-tuning (PEFT) has emerged as a core component of modern machine learning pipelines ( Houlsby et al. 2019 ; Han et al. 2024 ). Most PEFT methods adapt a frozen pre-trained backbone by learning a small set of task-specific parameters, often implemented as additive weight updates. Among these approaches, Low-Rank Adapters (LoRA) have become especially prominent, with a rapidly expanding literature ( Hu et al. 2022 ; Mao et al. 2025 ). Recent work has sought to further reduce the number of trainable parameters by exploring alternative low-rank decompositions ( Karimi Mahabadi et al. 2021 ; Edalati et al. 2022 ; Liu et al. 2024b ; He et al. 2023 ; Ding et al. 2023 ; Albert et al. 2025 ; Kopiczko et al. 2024 ; Koohpayegani et al. 2024 ). Recently, a new fine-tuning paradigm has emerged that enhances the expressive power of low-rank adapters by applying rank-enhancing functions component-wise. Introduced in Ji et al. 2025 , this approach demonstrates that applying a non-linear transformation, specifically, a fixed-frequency sinusoidal function, to a low-rank adapter can significantly increase its rank. Notably, this gain in expressivity comes at no additional parameter cost, preserving the efficiency of LoRA while yielding higher-rank representations. In this paper, we investigate the interaction between rank-enhancing sinusoidal non-linearities and quantization, a technique that maps full-precision parameters to a smaller set of discrete values, ideally with minimal impact on model performance ( Han et al. 2015 ; Gholami et al. 2021 ; Li et al. 2024a ). Quantization is a key enabler for deploying large models on resource-constrained hardware, offering improvements in memory efficiency, computational throughput, and energy consumption ( Gholami et al. 2021 ; Dettmers et al. 2024 ; Xu et al. 2024 ; Kaushal et al. 2025 ). To study this interaction, we develop a theoretical framework that characterizes how the rank of an adapter changes under quantization, showing that it is tightly controlled by the rank of the original, unquantized adapter. This leads to our key insight: when the adapter has low-rank, as is the case with LoRA, quantization preserves this structure. However, by applying a component-wise sinusoidal non-linearity after quantization, we can enrich the representational capacity of the adapter, effectively compensating for the rank limitation and enabling more expressive quantized models. This insight is particularly relevant in the context of adapter quantization, which has emerged as one of two dominant approaches in quantized fine-tuning. The first, exemplified by QLoRA ( Dettmers et al. 2023 ; Badri and Shaji 2024 ), applies quantization to the base model while maintaining high-precision adapters and activations. This approach is primarily motivated by reducing memory overhead during fine-tuning, making it feasible to adapt large language models on a single GPU ( Dettmers et al. 2024 ). The second approach focuses on quantizing the adapters themselves ( Yao and Klimovic 2023 ; Liu et al. 2024a ; Isik et al. 2023 ; Ping et al. 2024 ; Jie et al. 2023 ), enabling highly compact and transferable fine-tuned models. Our work follows this latter direction, showing that rank-enhancing sinusoidal functions can be easily integrated as a plug-in component into quantized adapters, significantly improving their expressivity while retaining the memory efficiency that makes adapter quantization attractive. To our knowledge, such rank-enhancing functions have not yet been explored in the quantization literature, and we view this work as a first step towards bridging that gap. Our main contributions are as follows: • We provide the first theoretical analysis showing how quantization affects the rank of a fine-tuning adapter, and show that this rank is tightly governed by the rank of its unquantized counterpart. • Based on our theoretical results we demonstrate that the effects of quantization on rank can be mitigated by applying rank enhancing functions in the form of sinusoids with fixed frequencies. We validate our approach through extensive experiments on vision and language tasks, including Large Language Model adaptation, Vision-Language Model Adaptation, and Text-to-Image Generation. For evaluation on Commonsense Reasoning, we achieve up to a 66% reduction in memory usage relative to full-precision LoRA models, without compromising performance. 2 Related Work Rank-Enhancing Functions. The most relevant rank-enhancing adapter works related to our approach are Ji et al. (2025) and Li et al. (2024) who investigate the use of sine-nonlinearities in low-rank adaptation Ji et al. 2025 ; Li et al. 2024b . We directly extend this approach by considering the effect of quantization on adapter performance. Parameter efficient adapters. Parameter efficient adaptation is a common fine-tuning strategy, in which a pre-trained base model is frozen, and a minimal number of adapter weights are trained on new data Houlsby et al. 2019 . Low-Rank Adapters are a common variant, in which the adapter comprises two low-rank matrices Hu et al. 2022 . VeRA Kopiczko et al. 2024 , RandLoRA Albert et al. 2025 , and NOLA Koohpayegani et al. 2024 use combinations of random projections to reduce the number of parameters contained within the adapters. QA-LoRA Xu et al. 2024 produces adapters that can be merged with the quantized base model, enabling low-precision inference. Delta compression and quantization. Although adapters represent a trivial proportion of the total number of parameters in a network (typically less than 1%), a recent branch of research has focused on the specific compression of these updates. Termed Delta Compression , this branch recognizes the practical importance of reducing the memory throughput of fine-tuned updates, which may be distributed at scale to many parties with a common base model Isik et al. 2023 ; Yao and Klimovic 2023 ; Brüel-Gabrielsson et al. 2025 . Within this framework it is typical to quantize the adapters, by mapping values to a limited set of floating points. This is often combined with lossless entropy

Read the complete text