Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA

Patryk MarszałekKlaudia BałazyJacek Tabor, Tomasz KuśmierczykJagiellonian University · 2025

Read Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA by Patryk MarszałekKlaudia BałazyJacek Tabor, Tomasz KuśmierczykJagiellonian University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12122.txt.

Introduction

Patryk Marszalek, Klaudia Balazy, Jacek Tabor, and Tomasz Kusmierczyk add uncertainty checks to a lean tuning method in 2025. The method tunes huge language models by training only small low-rank pieces. It is cheap, but it gives no sense of how sure the model is.

Probabilistic versions exist, yet they add many trainable parts and can be unstable. The new method does its uncertainty work in a tiny projected space instead. Tests show good accuracy with better calibration and wider coverage. The team also finds that the uncertainty patterns themselves have low rank, which supports the lean design.

Catalog record

CREATORPatryk MarszałekKlaudia BałazyJacek Tabor, Tomasz KuśmierczykJagiellonian University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12122

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA Patryk MarszałekKlaudia BałazyJacek Tabor, Tomasz KuśmierczykJagiellonian University Abstract Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large language models by decomposing weight updates into low-rank matrices, significantly reducing storage and computational overhead. While effective, standard LoRA lacks mechanisms for uncertainty quantification, leading to overconfident and poorly calibrated models. Bayesian variants of LoRA address this limitation, but at the cost of a significantly increased number of trainable parameters, partially offsetting the original efficiency gains. Additionally, these models are harder to train and may suffer from unstable convergence. In this work, we propose a novel parameter-efficient Bayesian LoRA, demonstrating that effective uncertainty quantification can be achieved in very low-dimensional parameter spaces. The proposed method achieves strong performance with improved calibration and generalization while maintaining computational efficiency. Our empirical findings show that, with the appropriate projection of the weight space: (1) uncertainty can be effectively modeled in a low-dimensional space, and (2) weight covariances exhibit low ranks. 1 1 footnotetext: Primary contributors. 1 Introduction LoRA (Low-Rank Adaptation) Hu et al. 2021 reduces computational overhead by decomposing the update weights of pre-trained models into low-rank matrices, enabling efficient adaptation to downstream tasks. Minimizing the number of trainable parameters reduces memory and storage requirements, making large-scale model adaptation feasible. Reducing computational overhead speeds up training time and makes adaptation possible in resource-constrained settings. Unlike pre-trained models, which are relatively well-calibrated ( OpenAI 2023 ) , fine-tuned large models (e.g., LLMs) often become overconfident and poorly calibrated ( Jiang et al. 2021 ; Tian et al. 2023 ; Xiao et al. 2022 ; He et al. 2023 ) , especially when trained on limited data. This hinders their usability for applications where uncertainty-aware decisions are performed. Figure 1: Performance averaged over multiple GLUE datasets (individual results in Fig. 3 ). Our method achieves superior calibration (ECE) and competitive predictive performance (Brier) while maintaining computational efficiency. For example, at r=8 ( \blacktriangleup ), we reduce ECE by half with only 1/10th LoRA parameters. Method r k # Parameters Standard LoRA 2 - 0.2M LoRA 8 - 0.8M LoRA-XS 8 - 6k LoRA-XS 25 - 60k Bayesian SWAG-LoRA 2 10 2.4M SWAG-LoRA 8 10 9.4M SWAG-LoRA 8 5 5.5M B-LoRA-XS 8 10 74k B-LoRA-XS 25 10 0.7M B-LoRA-XS 25 5 0.4M Figure 2: (left): Weight-adaptation approaches: LoRA vs. B-LoRA-XS. As indicated by the color coding, some parameters remain frozen ( blue ), others are trained ( orange ) or obtained via SVD ( green ). (right): Number of trainable parameters per method. XS variants remain computationally competitive even for ranks as large as r=25 . Bayesian treatment is then frequently proposed to address overconfidence in neural networks ( Blundell et al. 2015 ; Kristiadi et al. 2020 ; Aitchison et al. 2021 ; Izmailov et al. 2021 ) . Consequently, recently proposed Bayesian variants of LoRA ( Onal et al. 2024 ; Robeyns 2024 ; Doan et al. 2025 ) address the aforementioned challenges by introducing uncertainty estimation directly into the fine-tuning process. During training, these models continuously adjust both the mean and covariance of fine-tuned parameters to achieve better generalization and uncertainty quantification. Learning the posterior covariance matrix is necessary for modeling epistemic uncertainty. However, its size grows quadratically with the number of parameters, which can easily cancel out the benefits of LoRA, in addition to making learning significantly harder. Using low-rank, Kronecker-factored, or diagonal-only covariances partially alleviates the problem, but as we demonstrate in Sec. 3 , this comes at the cost of results quality loss. Furthermore, even at rank = 2, the number of trainable parameters is quadrupled compared to vanilla LoRA. This creates a need for an alternative approach that retains covariance modeling capacity while reducing the number of required parameters. We propose a method that learns Bayesian posteriors for weights projected onto a low-dimensional manifold, hence maintaining parameter efficiency. The thoughtfully selected projection allows for the effective representation of the covariances between weights through covariances between representations in the lower-dimensional space. In this design, we follow the work of Bałazy et al. 2024 , who recently proposed a strategy for finding such projections with SVD. We prove that they are effective for learning Bayesian models as well. Operating in such a reduced parameter space significantly improves the feasibility of Bayesian inference. We show that correlations between weights can be represented very efficiently – unlike in the original weight space, we can use covariance matrices with ranks as low as k=2 . Thanks to the low number of parameters, training is also more stable. Finally, the method achieves superior calibration and accuracy at low budgets (e.g., see Fig. 1 ). In the Appendix, we supplement the results presented in the paper with a detailed overview of the experimental setup. We make our code publicly available at github . CoLA MRPC RTE SST-2 Figure 3: Median \pm std. accuracy (left), ECE (middle), and NLL (right) on 4 GLUE tasks (rows) vs. total parameter count for several methods and varying ranks r . B-LoRA-XS (our) achieves the accuracy and the calibration of SWAG-LoRA while using significantly fewer parameters than LoRA. See Fig. 1 for averaged results. 2 Method LoRA fine-tunes large pre-trained models by learning low-rank weight updates \Delta W instead of training the weights W directly. For a pre-trained parameter matrix W^{0}\in\mathbb{R}^{m\times n} that is kept fixed, LoRA learns a rank- r update \Delta W=AB , where A\in\mathbb{R}^{m\times r} and B\in\mathbb{R}^{r\times n} have far fewer parameters. The effective weight is then: W=W^{0}+\Delta W=W^{0}+AB, where only A and B are trained. Typically, LoRA is applied jointly at multiple layers, yielding a set of updates \{\Delta W_{l}\} . Bayesian treatment of a neural network involves finding the posterior p(\theta\mid\mathcal{D}) given training data \mathcal{D} . By Bayes’ theorem: p(\theta\mid\mathcal{D})\;=\;\frac{p(\mathcal{D}\mid\theta)\,p(\theta)}{p(\mathcal{D})}, where \theta represents the model’s parameters (i.e., weights) considered random variables. In particular, for the Bayesian LoRA setting, \theta denotes a set of the learned model updates , while the remaining frozen weights are hidden inside the model likelihood, given by p(\mathcal{D}\mid\theta)=\prod_{i\in[\mathcal{D}]}p(y_{i}|x_{i},\theta) . The learned posterior allows Bayesian model averaging at inference as: p(y_{*}\mid x_{*},\mathcal{D})\;=\;\int p(y_{*}\mid x_{*},\theta)\,p(\theta\mid\mathcal{D})\,d\theta\approx\frac{1}{S}\sum_{\theta\sim p(\theta|D)}p(y_{*}\mid x_{*},\theta), where we use S=15 samples from the posterior. Bayesian LoRAs obtain the posterior for \{\Delta W_{l}\} through the learned posterior for \theta=\cup_{l}\{A_{l}\cup B_{l}\} , where l indexes the weight updates (layers). The posterior itself is approximated either using a set of particles or a closed-form distribution. Due to its superior performance, we rely on the latter and assume p(\theta|\mathcal{D})\approx\mathcal{N}(\mu,\Sigma) , where \mu is the vector of means (of size equal to the number of learned parameters) and \Sigma is the covariance matrix, whose size grows quadratically with the total number of parameters. Notably, we aim to model cross-layer interdependencies , requiring covariance estimation also across weights in different layers \{l\} . However, this results in an impractically large number of parameters. Consequently, we explore methods to reduce this cost by representing distributions p(\{\Delta W\}|\mathcal{D}) differently. In LoRA-XS Bałazy et al. 2024 , the adaptation matrices A and B are initialized using the truncated SVD of the corresponding pre-trained weight matrices W^{0} . This initialization captures the most informative singular components of the original weights. Under the assumption that the fine-tuned task is

Read the complete text