MatryoshkaLora : Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
Ionut-Vlad Modoranu, Mher Safaryan, Dan Alistarh · 2026
Read MatryoshkaLora : Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning by Ionut-Vlad Modoranu, Mher Safaryan, Dan Alistarh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07850.txt.
Introduction
Ionut-Vlad Modoranu and two co-authors remove the rank-guessing problem from LoRA, the standard low-rank method for fine-tuning large models cheaply. LoRA's rank must be fixed in advance, and finding a good one takes exhaustive search. Rank-adaptive variants sample ranks during training but learn the high ranks poorly, because gradient signals are inconsistent across the hierarchy.
Their MatryoshkaLoRA inserts a fixed, carefully crafted diagonal matrix between the LoRA factors so every sub-rank receives the gradient information efficiently, nesting ranks like the dolls the method is named for. Rank can then be chosen dynamically with minimal accuracy loss. They also propose a metric, the area under the rank-accuracy curve, and show better accuracy-efficiency trade-offs than earlier adaptive methods.
Catalog record
| CREATOR | Ionut-Vlad Modoranu, Mher Safaryan, Dan Alistarh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07850
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MatryoshkaLora : Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning Ionut-Vlad Modoranu, Mher Safaryan, Dan Alistarh Abstract With the rise in scale for deep learning models to billions of parameters, the computational cost of fine-tuning remains a significant barrier to deployment. While Low-Rank Adaptation (LoRA) has become the standard for parameter-efficient fine-tuning, the need to set a predefined, static rank r r requires exhaustive grid searches to balance efficiency and performance. Existing rank-adaptive solutions such as DyLoRA mitigate this by sampling ranks during the training from a predefined distribution. However, they often yield sub-optimal results at higher ranks due to lack of consistent gradient signals across the full hierarchy of ranks, thus making these methods data-inefficient. In this paper, we propose MatryoshkaLoRA , a general, Matryoshka-inspired training framework for LoRA that learns accurate hierarchical low-rank representations by inserting a fixed, carefully crafted diagonal matrix P P between the existing LoRA adapters to scale their sub-ranks accordingly. By introducing this simple modification, our general framework recovers LoRA and DyLoRA only by changing P P and ensures all sub-ranks embed the available gradient information efficiently. Our MatryoshkaLoRA supports dynamic rank selection with minimal degradation in accuracy. We further propose Area Under the Rank Accuracy Curve (AURAC), a metric that consistently evaluates the performance of hierarchical low-rank adapters. Our results demonstrate that MatryoshkaLoRA learns more accurate hierarchical low-rank representations than prior rank-adaptive approaches and achieves superior accuracy-performance trade-offs across ranks on the evaluated datasets. Our code is available at https://github.com/IST-DASLab/MatryoshkaLoRA . 1 Introduction & Related Work Figure 1: MatryoshkaLora The research advancements in the Deep Learning community allowed researchers and practitioners to train models with billions of parameters. Building and managing pipelines for such a complex system requires a significant engineering effort, from preparing the dataset to scaling the model and the optimizer states via multi-dimensional parallelization techniques across many GPUs in a cluster or even multiple physical clusters - a process spanning over months. Given the costs, these large models are rarely deployed to production-ready environments as they are; instead, they serve as a knowledge base for downstream adaptation, such as fine-tuning, which still remains computationally prohibitive for many applications and settings. To alleviate this overhead, LoRA 9 has emerged as the de-facto standard for parameter-efficient fine-tuning (PEFT). However, it introduces a significant structural constraint: the rank r must be predefined. Finding the optimal balance between parameter efficiency and model performance currently requires exhaustive search across multiple training runs, each with a different static rank. One line of work studies adaptive-rank LoRA methods, which optimize how rank capacity is allocated across layers or modules under a given or learned parameter budget. These methods improve parameter efficiency, but they typically produce a single specialized adapter configuration rather than a nested adapter family that can be sliced at inference time 23 ; 6 ; 14 ; 24 ; 22 ; 17 ; 5 ; 11 . A second direction studies dynamic-rank LoRA methods, which aim to train a single adapter whose prefixes are independently usable as lower-rank adapters. Unlike adaptive allocation methods, these approaches are designed to provide multiple sub-adapters from one checkpoint 18 ; 16 . A conceptually related, but not directly comparable, line of work is slimmable 20 and once-for-all networks 2 . These are not LoRA methods, but they provide the broader once-for-all training paradigm: a single set of weights is trained so that many nested sub-networks are valid deployment choices 19 . Our work can be interpreted as translating this paradigm from channel width to LoRA rank and belongs primarily to the second cluster. It differs from adaptive-rank allocation methods because it does not merely identify one rank configuration; it trains a nested adapter family. It differs from general slimmable networks because the nested structure is imposed on low-rank adapter factors rather than full-model channels. Its closest comparison is DyLoRA 18 , which also targets dynamic rank usage but enforces rank hierarchy through a different training mechanism discussed next. DyLoRA 18 is an alternative solution to grid-searching for the rank r for LoRA. At each training step, a rank k\in\mathbb{N} is randomly sampled from a predefined distribution and the forward pass is performed using only the first k rows and first k columns of adapters A and B , respectively, denoted by A_{k} and B_{k} . The loss is computed with respect to the output of the network that was generated using only A_{k} and B_{k} for all linear layers. The purpose of this approach is to train adapters A and B that still yield high accuracy when using only a slice of size k on the fly. DyLoRA solves the issue of adaptive rank r only partially because, as we will show in our experimental section, the accuracies of the models fine-tuned with DyLoRA do not exhibit a true hierarchical pattern in the context of reasoning tasks, such as math datasets. We empirically show that DyLoRA strategy is sub-optimal because it constrains the learning only to a fixed rank k sampled at random, while the ranks r>k do not receive any gradient signal. Our work proposes Matryoshka-style training framework for LoRA, inspired from Matryoshka Representation Learning 12 , which we call MatryoshkaLora . It is motivated by the drawback of DyLoRA, which fails to learn what we call hierarchical low-rank features. Instead of using only one randomly generated rank k per forward pass, we use all possible slices A_{r} and B_{r} , with r\in\{1,2,4,...,R\} , where R is the maximum rank of A and B . This way, the lower-dimensional representations are contained in higher-dimensional representations as prefixes, thus building a nested hierarchy of features. We present the schematic of MatryoshkaLora in Figure 1 . We see three advantages of having an accurate technique to train hierarchical low-rank adapters: (1) it eliminates the need for multiple training runs to grid-search for different ranks, implicitly translating to a reduction in costs; (2) we benefit from a high-performance adapter at each rank r by deploying targeted ranks on different devices depending on their computational power and (3) we enable dynamic rank selection under varying cluster loads to serve the requests at the same rate with minimal accuracy drop. Contribution. We summarize our contribution as follows: • We introduce a general framework inspired by Matryoshka Representation Learning for training LoRA adapters whose lower-rank prefixes remain accurate and independently usable at inference time, which views LoRA, DyLoRA, and Matryoshka-style adapters as different parameterizations of a shared rank-weighting vector; • Starting from the goal of learning hierarchical low-rank representations at every layer, we show that this naturally leads to a simple diagonal weighting between the standard LoRA adapters A and B , making the hierarchy explicit while keeping the implementation and exposition close to standard LoRA; • We propose MatryoshkaLora , a concrete instance of this framework that learns accurate nested low-rank prefixes via the new diagonal matrix. The resulting adapter can be evaluated or deployed at multiple ranks from a single checkpoint, without retraining separate adapters; • We introduce Area Under the Rank Accuracy Curve (AURAC) , a metric for evaluating adapter performance across a set of ranks. AURAC summarizes the rank-performance tradeoff while weighting each rank according to its magnitude, reflecting the