CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
Zhuxuanzi Wang Mingqiao Mo Xi Xiao Chen Liu Chenrui Ma Yunbei Zhang Xiao Wang Smita Krishnaswamy Tianyang Wang · 2025
Read CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models by Zhuxuanzi Wang Mingqiao Mo Xi Xiao Chen Liu Chenrui Ma Yunbei Zhang Xiao Wang Smita Krishnaswamy Tianyang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-15962.txt.
Introduction
Zhuxuanzi Wang leads eight co-authors with a low-rank tuning method that decides where capacity goes by curvature, not by habit. Low-rank adaptation updates a small set of matrices to specialise a large model cheaply. Earlier methods, though, set the update budget apart from how training actually evolves, which wastes capacity or destabilises runs.
CTR-LoRA schedules rank by marginal utility estimated from light second-order proxies, so useful directions gain room first. A trust region built from Fisher and Hessian metrics then keeps each update stable. Across several open backbones in the seven to thirteen billion parameter range, on both familiar and shifted benchmarks, the method beats strong baselines consistently. Accuracy rises while memory use falls and throughput grows, placing the method on the performance and efficiency frontier, the team reports.
Catalog record
| CREATOR | Zhuxuanzi Wang Mingqiao Mo Xi Xiao Chen Liu Chenrui Ma Yunbei Zhang Xiao Wang Smita Krishnaswamy Tianyang Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-15962
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models Zhuxuanzi Wang Mingqiao Mo Xi Xiao Chen Liu Chenrui Ma Yunbei Zhang Xiao Wang Smita Krishnaswamy Tianyang Wang Abstract Parameter-efficient fine-tuning (PEFT) has become the standard approach for adapting large language models under limited compute and memory budgets. Although previous methods improve efficiency through low-rank updates, quantization, or heuristic budget reallocation, they often decouple the allocation of capacity from the way updates evolve during training. In this work, we introduce CTR-LoRA, a framework guided by curvature trust region that integrates rank scheduling with stability-aware optimization. CTR-LoRA allocates parameters based on marginal utility derived from lightweight second-order proxies and constrains updates using a Fisher/Hessian-metric trust region. Experiments on multiple open-source backbones (7B-13B), evaluated on both in-distribution and out-of-distribution benchmarks, show consistent improvements over strong PEFT baselines. In addition to increased accuracy, CTR-LoRA enhances training stability, reduces memory requirements, and achieves higher throughput, positioning it on the Pareto frontier of performance and efficiency. These results highlight a principled path toward more robust and deployable PEFT. † † address: 1 Cornell University 2 University of Alabama at Birmingham 3 Oak Ridge National Laboratory 4 Yale University 5 University of California, Irvine 6 Tulane University ∗ Equal contribution † Corresponding author: [email protected], [email protected] 1 Introduction Parameter-efficient fine-tuning (PEFT) has become the standard route to adapt billion-parameter language models under strict compute and memory budgets [ 1 , 2 , 3 , 4 , 5 ] . Among PEFT designs, low-rank adaptation (LoRA) injects a pair of thin adapters into selected projections and freezes the backbone, achieving strong transfer with no inference-time latency and a minimal trainable footprint [ 6 ] . Quantization-aware PEFT pushes this boundary further: QLoRA enables 4-bit training and makes even 65B models attainable on a single 48 GB GPU [ 7 ] . Despite its ubiquity, LoRA still faces two pain points: (1) rank schedules are typically uniform or hand-tuned, ignoring the heterogeneous importance of layers and matrices, and (2) training stability depends on update geometry and step size, leaving low-rank updates vulnerable in high-curvature regions. Recent studies also highlight pronounced hyperparameter sensitivity and a trade-off between acquiring new skills and retaining base knowledge [ 8 ] . Figure 1: Comparison between LoRA and CTR-LoRA. Left: standard LoRA decomposes the low-rank update \Delta W=AB^{\top} with A\sim\mathcal{N}(0,\sigma^{2}) and B=0 , merged into frozen pretrained weights W\in\mathbb{R}^{d\times d} . Right: CTR-LoRA introduces curvature-aware rank scheduling (guiding the choice of rank r ) and a trust-region penalty that regularizes the update trajectory, improving stability and robustness during fine-tuning. A series of variants tackle these challenges from different angles. Adaptive-budget methods redistribute rank across matrices (e.g., AdaLoRA) under a global parameter cap [ 9 ] . Optimization-aware refinements adjust the update rule: DoRA decouples magnitude and direction [ 10 ] , while LoRA+ applies asymmetric learning rates to correct imbalanced dynamics [ 11 ] . Initialization-aware schemes such as PiSSA seed adapters with principal components to accelerate convergence [ 12 ] . More recently, sensitivity-driven allocation leverages second-order signals for rank selection with low overhead (Sensitivity-LoRA), underscoring the value of curvature information [ 13 ] . Yet most methods still optimize rank separately from training dynamics: they decide where to place parameters but not how low-rank updates evolve across the loss landscape. This work proposes a purely adapter-based solution that unifies where to allocate capacity with how to move in the parameter space. We introduce a curvature-informed, trust-region-guided framework that (1) scores marginal utility of rank-1 directions under a global budget for rank scheduling, and (2) regularizes training with a curvature-metric penalty to keep updates stable in high-curvature regions. The approach is plug-and-play with existing LoRA stacks (compatible with LoRA+, DoRA, and 4-bit training), incurs zero inference-time cost, and preserves the simplicity practitioners expect. Our contributions are summarized as follows: • We introduce a principled criterion based on marginal utility under curvature proxies, replacing uniform or heuristic schedules. This shifts PEFT from manual tuning toward theoretically guided allocation, setting a foundation for more systematic use of limited parameters. • We establish a stability-oriented objective that constrains updates in curvature space, ensuring reliable optimization and quantization compatibility. This provides a unifying perspective that links PEFT to trust-region and natural-gradient methods, bridging practice with theory. • Our framework is fully compatible with existing LoRA ecosystems, requires no architectural changes, and preserves zero inference overhead. This positions CTR-LoRA not merely as another variant, but as a deployable and extensible paradigm for robust, large-scale adaptation. Figure 2: Comprehensive analyses of CTR-LoRA. (a-b) Budget-performance curves show consistent improvements. (c) CTR-LoRA has smoother training dynamics and converges to higher accuracy. (d) K-FAC and Hutchinson proxies outperform sensitivity-only allocation, confirming robustness to curvature estimators. (e) Performance is stable within a moderate trust-region range. (f) CTR-LoRA occupies the Pareto frontier by reducing variance, lowering memory, and increasing throughput. 2 Method Problem Definition We adapt a pretrained transformer with parameter matrices \{W_{\ell}\}_{\ell=1}^{L} using low-rank adapters. For each target matrix W_{\ell}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}} , LoRA parameterizes an update \Delta W_{\ell}=A_{\ell}B_{\ell}^{\top} with rank r_{\ell}\ll\min(d_{\text{in}},d_{\text{out}}) and injects W_{\ell}\leftarrow W_{\ell}+\Delta W_{\ell} while freezing the backbone [ 6 ] . We assume a global trainable-parameter budget B=\sum_{\ell}r_{\ell}(d_{\text{in}}+d_{\text{out}}) , or equivalently a fixed ratio of base parameters. The central goal of CTR-LoRA is to allocate ranks such that each parameter provides the highest marginal reduction in loss, while simultaneously constraining the update geometry to remain stable in high-curvature regions, all without incurring inference-time latency. Curvature Proxies Second-Order Signals Let \mathcal{L} denote the training loss and G_{\ell}=\nabla_{W_{\ell}}\mathcal{L} the gradient. To capture local geometry, we approximate curvature with a block-diagonal proxy M_{\ell}\succeq 0 for each matrix. In practice, we employ either Kronecker-factored approximations (K-FAC) [ 14 ] , which decompose the Fisher block as \mathbb{E}[gg^{\top}]\otimes\mathbb{E}[aa^{\top}] based on layer inputs a and backpropagated gradients g , or Hutchinson-style estimators [ 15 , 16 ] , which use stochastic Hessian-vector products to approximate diagonals and quadratic forms. These statistics are precomputed on a small calibration set and refreshed intermittently during training. With M_{\ell} in place, we define the whitened gradient \tilde{G}_{\ell}=M_{\ell}^{-1/2}G_{\ell}M_{\ell}^{-1/2} , which normalizes directions by curvature and aligns with natural-gradient intuition [ 17 ] . Curvature-Aware Rank Scheduling We evaluate candidate rank-one directions using a local quadratic model. The second-order expansion at W_{\ell} is given by Eqn ( 1 ). \displaystyle\begin{split}\Delta\mathcal{L}&\approx-\langle G_{\ell},\Delta W_{\ell}\rangle+\tfrac{1}{2}\|\Delta W_{\ell}\|_{M_{\ell}}^{2}\\ \qquad\|X\|_{M_{\ell}}^{2}&=\langle X,M_{\ell}X\rangle\end{split} (1) Confining \Delta W_{\ell} to a form \alpha UV^{\top} with \|U\|_{2}=\|V\|_{2}=1 , the optimal step size is \alpha^{\star}=\langle G_{\ell},UV^{\top}\rangle/\|UV^{\top}\|_{M_{\ell}}^{2} , yielding a predicted decrease as shown in Eqn ( 2 ). \displaystyle\begin{split}\mathcal{D}_{\ell}(U,V)&=\tfrac{1}{2}\frac{\langle G_{\ell},UV^{\top}\rangle^{2}}{\|UV^{\top}\|_{M_{\ell}}^{2}}\\ &=\tfrac{1}{2}\,\sigma^{2}(M_{\ell}^{-1/2}G_{\ell}M_{\ell}^{-1/2})\end{split} (2) Here, the equality holds when (U,V) corresponds to a singular pair of the whitened gradient. Thus the leading singular vectors of \tilde{G}_{\ell} naturally provide descending marginal utilities. To respect the global budget B , we collect candidate directions across layers and select them greedily by their marginal utility scores. Each selection assigns one unit of rank, optionally followed by rank-one deflation to update the residual spectrum. In practice, randomized SVD [ 18 ] with small oversampling suffices to approximate these spectra efficiently. Compared with uniform or hand-tuned schedules, this strategy explicitly favors matrices where curvature-adjusted gradients carry the most useful information. Trust-Region PEFT Objective Rank allocation alone does not constrain how