LoCO: Low-rank Compositional Rotation Fine-tuning

An Nguyen, Jaesik Choi, Anh Tong · 2026

Read LoCO: Low-rank Compositional Rotation Fine-tuning by An Nguyen, Jaesik Choi, Anh Tong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15916.txt.

Introduction

An Nguyen, Jaesik Choi and Anh Tong introduce LoCO, Low-rank Compositional Orthogonal fine-tuning for large base models. Existing low-rank methods gain parameter savings through low-rank weight updates but struggle to preserve pretrained feature geometry. LoCO builds orthogonal maps from low-rank skew-symmetric matrices and rotation chains, with an approximation scheme for parallel rotation math. The method keeps low compute cost while keeping orthogonality with bounded error. Tests across diffusion, vision and language models show top or matching results against orthogonal rivals.

Catalog record

CREATORAn Nguyen, Jaesik Choi, Anh Tong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61474
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15916

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LoCO: Low-rank Compositional Rotation Fine-tuning An Nguyen, Jaesik Choi, Anh Tong Abstract Parameter-efficient fine-tuning (PEFT) has emerged as an critical technique for adapting large-scale foundation models across natural language processing and computer vision. While existing methods such as low-rank adaptations achieve parameter efficiency via low-rank weight updates, they are limited in their ability to preserve the geometric structure of pretrained representations. We introduce Low-rank Compositional Orthogonal fine-tuning (LoCO), a novel PEFT method that constructs orthogonal transformations through low-rank skew-symmetric matrices and compositional rotation chains. We propose an approximation scheme that enables fully parallel computation of compositional rotations, making the approach practical for high-dimensional feature spaces. Our method maintains low computational complexity while maintaining orthogonality with controlled approximation error. We validate LoCO across diverse domains, including diffusion transformer fine-tuning, vision transformer adaptation, and language model adaptation. Our method demonstrates superior or competitive performance compared to both existing orthogonal and non-orthogonal methods. 1 Introduction Large-scale pretrained foundation models Vaswani et al. 2017 have revolutionized computer vision and natural language processing, achieving impressive performance across diverse downstream tasks. Models such as large language models Brown et al. 2020 , vision transformers Dosovitskiy 2020 and diffusion transformers Rombach et al. 2022 ; Peebles and Xie 2023 contain billions of parameters, encoding rich representations learned from massive datasets. Adapting these models to specific applications through full fine-tuning is often impractical: it demands substantial computational resources, risks catastrophic forgetting of pretrained knowledge, and requires storing separate model copies for each task. Parameter-efficient fine-tuning (PEFT) methods Houlsby et al. 2019 ; Hu et al. 2022 have emerged as an effective solution for task adaptation while training only a small fraction of parameters. Orthogonal fine-tuning methods Qiu et al. 2023 ; Liu et al. 2024b ; Yuan et al. 2024 have attracted attention for their geometric properties. Unlike additive methods such as LoRA Hu et al. 2022 that injects low-rank adaptation to weight matrices, orthogonal methods apply multiplicative transformations, which are constrained to the special orthogonal group. This constraint offers a theoretical advantage in that orthogonal transformations preserve the pairwise angular relationships among neuron representations and maintain the hyperspherical energy structure of pretrained features Qiu et al. 2023 . Such properties are particularly valuable when semantic information is encoded in angular relationships rather than magnitudes, as observed in contrastive learning Chen et al. 2020 and face recognition Liu et al. 2017 . Despite these properties, existing orthogonal fine-tuning methods face a fundamental computational bottleneck. Constructing and applying a general orthogonal matrix requires cubic time complexity for matrix inversion or matrix exponential computation, which becomes prohibitively expensive for high-dimensional feature spaces typical in modern architectures. Prior work addresses this challenge through structural constraints. For example, OFT Qiu et al. 2023 uses block-diagonal matrices that restrict rotations to independent subspaces, while BOFT Liu et al. 2024b uses butterfly factorizations for hierarchical decomposition. However, these approaches impose specific sparsity patterns that may limit expressivity, and butterfly structures may suffer from non-contiguous memory access patterns that underutilize hardware resources. In this paper, we propose Lo w-rank C ompositional O rthogonal Fine-tuning ( LoCO ), a novel approach that constructs orthogonal transformations through low-rank skew-symmetric matrices. Our key observation is that the tangent space to the special orthogonal group, which is the space of skew-symmetric matrices, admits a low-rank parameterization. By exploiting this structure with the Sherman–Morrison–Woodbury identity Max 1950 , we reduce the computational complexity of orthogonal fine-tuning, allowing adaptation in high-dimensional spaces. Additionally, we leverage chains of multiple low-rank rotations to enhance the expressiveness of orthogonal transformations. Rather than computing these rotations sequentially, we employ a first-order approximation that enables parallel computation across all rotation components. Our analysis confirms that this approximation introduces negligible error given the small-magnitude adaptations characteristic of fine-tuning. A practical advantage of LoCO is a test-time temperature scaling mechanism for controllable adaptation strength. By introducing a scalar temperature parameter, we can smoothly interpolate between pretrained and fully-adapted behaviors at inference time. We test LoCO across three domains: (1) fine-tuning language models, (2) adapting diffusion transformers for conditional generation, and (3) fine-tuning vision transformers. Across all settings, LoCO achieves competitive or superior performance compared to established PEFT methods including LoRA Hu et al. 2022 , OFT Qiu et al. 2023 , BOFT Liu et al. 2024b , and HRA Yuan et al. 2024 , while maintaining computational efficiency and a low memory footprint. Contributions. Our main contributions are threefold: • We propose LoCO, a parameter-efficient fine-tuning method that constructs orthogonal transformations via low-rank skew-symmetric matrices. We further introduce an approximation for compositional rotation chains that enables parallel computation while preserving orthogonality for fine-tuning. • We demonstrate a test-time temperature scaling mechanism that enables flexible control over adaptation strength without retraining, providing an alternative to costly regularization schemes used in prior orthogonal methods. • We achieve state-of-the-art or competitive results across diverse benchmarks, including GLUE, mathematical reasoning, and controllable generation. 2 Related Work Low-rank Adaptation. Building on the low intrinsic rank hypothesis Aghajanyan et al. 2020 , LoRA Hu et al. 2022 parameterizes weight updates as a product of two low-rank matrices. This avoids the inference overhead of adapters Houlsby et al. 2019 and the optimization challenges in prompt tuning Li and Liang 2021 ; Lester et al. 2021 . Subsequent works extend this framework through adaptive rank allocation Zhang et al. 2023 ; Kalajdzievski 2023 ; Jiang et al. 2024 , alternative initializations Meng et al. 2024 ; Wang et al. 2024 , and parameter sharing Bałazy et al. 2024 ; Dong et al. 2023 ; Kopiczko et al. 2024 . DoRA Liu et al. 2024a further decomposes updates into magnitude and direction components. Unlike this line of research, our approach applies multiplicative transformations, constructing rotation matrices from low-rank skew-symmetric parameterizations. Orthogonal Fine-Tuning. OFT Qiu et al. 2023 ; Qiu et al. 2025 parameterizes orthogonal matrices through block-diagonal, butterfly structures Liu et al. 2024b , structured matrices Gorbunov et al. 2024 , exploiting sparsity for efficiency. While qGOFT Ma et al. 2024 decomposes rotations into Givens rotations, its sequential composition limits parallelization. Householder-based methods such as Dong et al. 2024 ; Yuan et al. 2024 ; Arcas et al. 2025 use reflection parameterizations. Recent work Liao and Monz 2024 applies 2D rotations directly to representation subspaces. Our method differs by constructing rotations from low-rank skew-symmetric matrices, enabling efficient computation through first-order approximation while operating on higher-dimensional subspaces. Spectral-based Methods. In another line of research, PiSSA Meng et al. 2024 and MiLoRA Wang et al. 2024 decompose weights via singular value decomposition. These approaches adapt residual or minor singular components selectively. Compared to these spectral-based methods, our approach preserves angular structure through orthogonal transformations rather than modifying spectral components. Geometric Approaches. Several works exploit geometric structures: hyperbolic networks Ganea et al. 2018 ; Chen et al. 2022 utilize exponential maps for hierarchical representations, and Lie group methods Mironenco and Forré 2024 ; Finzi et al. 2020 construct equivariant architectures. Skew orthogonal convolutions Singla and Feizi 2021 use matrix exponentials for training stability. While building from similar mathematical tools, these works focus on designing new architectures rather than proposing new PEFT methods. Figure 1: An overview of LoCO. We parameterize each rotation component \mathbf{R}_{i} using low-rank skew-symmetric matrices constructed from

Read the complete text