Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models
Liyi Zhang, Jake Snell, Thomas L. Griffiths · 2025
Read Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models by Liyi Zhang, Jake Snell, Thomas L. Griffiths (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14285.txt.
Introduction
Liyi Zhang, Jake Snell, and Thomas Griffiths introduce a meta-learning method for adapting large language models, in two thousand twenty-five. A cheap tuning method adjusts a model with small added matrices for one data set. When a task needs learning from many data sets at once, that approach generalizes poorly and costs rise.
Their method learns how to adapt, in a Bayesian way that spreads cost across tasks. It reframes which parts count as local or global in the small matrices, with a knob to balance fit against faith to shared settings. It scaled to large open models and beat rivals on two question-answering benchmarks. Joining it with in-context examples helped further.
Catalog record
| CREATOR | Liyi Zhang, Jake Snell, Thomas L. Griffiths |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-14285
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models Liyi Zhang, Jake Snell, Thomas L. Griffiths Abstract Fine-tuning large language models (LLMs) with low-rank adaptaion (LoRA) is a cost-effective way to incorporate information from a specific dataset. However, it is often unclear how well the fine-tuned LLM will generalize, i.e., how well it will perform on unseen datasets. Methods have been proposed to improve generalization by optimizing with in-context prompts, or by using meta-learning to fine-tune LLMs. However, these methods are expensive in memory and computation, requiring either long-context prompts or saving copies of parameters and using second-order gradient updates. To address these challenges, we propose Amortized Bayesian Meta-Learning for LoRA (ABMLL). This method builds on amortized Bayesian meta-learning for smaller models, adapting this approach to LLMs while maintaining its computational efficiency. We reframe task-specific and global parameters in the context of LoRA and use a set of new hyperparameters to balance reconstruction accuracy and the fidelity of task-specific parameters to the global ones. ABMLL provides effective generalization and scales to large models such as Llama3-8B . Furthermore, as a result of using a Bayesian framework, ABMLL provides improved uncertainty quantification. We test ABMLL on Unified-QA and Crossfit datasets and find that it outperforms existing methods on these benchmarks in terms of both accuracy and expected calibration error. 1 Introduction Large language models (LLMs) handle a variety of tasks reasonably well ( Radford et al. 2019 ) . However, to tailor LLMs to specific domains, fine-tuning on specific datasets is often necessary. While methods such as low-rank adaptation (LoRA; Hu et al. 2021 ) fine-tune a pretrained LLM cost-effectively, a fine-tuned LLM is limited to the domain it is trained on. Its performance may not improve in other domains and sometimes worsens as it suffers from catastrophic forgetting. Such catastrophic forgetting may result in overfitting and erasing existing capabilities of the pretrained LLM ( Lazaridou et al. 2021 ; Luo et al. 2023 ) . Meta-learning is a strategy for solving this problem, training models on a variety of tasks in a way that supports generalization across tasks ( Finn et al. 2017 ) . However, meta-learning typically requires a large amount of computation and memory, making it challenging to apply to LLMs. One form of meta-learning that has been applied to LLMs involves fine-tuning models on in-context prompt-response examples ( Min et al. 2022 ; Chen et al. 2022 ) . Another more traditional approach, MAML-en-LLM ( Sinha et al. 2024 ) , adapts the Model-Agnostic Meta-Learning (MAML) ( Finn et al. 2017 ) framework to LLMs. However, both methods are limited in the size of the language models that can be used: the former requires long-context prompts, whereas the latter uses second-order gradient updates and saves a model for each task. Recent work on Amortized Bayesian Meta-Learning (ABML; Ravi and Beatson 2019 ) addresses some of the computation and memory requirements of meta-learning. This approach posits a generative model over parameters where task-specific parameters are generated from global parameters, and inference over task-specific parameters is amortized. In other words, the conditional distribution over task-specific parameters is shared across tasks, implying that computation and memory costs stay constant with respect to the number of tasks. This approach thus offers a path towards efficient meta-learning for LLMs. However, several challenges exist. First, we need to specify the generative model over weight space in the context of LLMs. Second, the prior term used in ABML no longer adapts to the setting of fine-tuning a pretrained model because the spread of its weights mismatches that of an arbitrary prior used to train a model from scratch. Third, the enormous size of LLMs makes training difficult, as the scale of probabilities assigned to the model variables can overwhelm the influence of the data likelihood. In this paper, we present a solution to these problems, taking a Bayesian approach to fine-tuning LLMs using ABML. To define the underlying generative model and efficiently characterize the distributions involved, we use LoRA to express both the model weights and their uncertainty. We introduce a new prior over global variables that accounts for the spread of the parameters learned in the pretrained model. We also introduce two adjustable hyperparameters that balance reconstruction accuracy and the fidelity of task-specific parameters to the global ones. Using amortized Bayesian meta-learning for LLM fine-tuning, we achieve both higher accuracy and better uncertainty estimation over unseen tasks compared with regular fine-tuning and other scalable methods in the meta-learning literature. Figure 1 illustrates an example where incorporating uncertainty estimation in fine-tuning leads to a more calibrated model response. Our method is scalable and avoids the computation and memory overhead of other meta-learning approaches, making it adaptable to larger models such as Llama 3 8B . We show that amortized Bayesian meta-learning provides fine-tuned LLMs that are accurate on domain-specific tasks, more generalizable to new tasks, and provide better uncertainty estimation. Figure 1: An example where better uncertainty calibration leads to a more reasonable response. This is a prompt and response from an unseen dataset, coming from a pretrained LLM (left) and an LLM fine-tuned with ABMLL (right), with both being updated with 10 gradient steps on other examples of this dataset as in the meta-learning literature. The label is B), so both LLMs are incorrect, but the question is ambiguous: it could interpreted as either Jason “asked" Steven to deliver, or Jason “came to" Steven to deliver, resulting in different answers. ABMLL results in a more calibrated response. 2 Related Work Meta-learning methods in LLMs. Extensive work has explored meta-learning for generalization, typically adopted for models in the pre-LLM era ( Finn et al. 2017 ; Snell et al. 2017 ; Ravi and Beatson 2019 ; Nichol et al. 2018 ) . Sinha et al. 2024 adapted Model-Agnostic Meta-Learning (MAML), developed in Finn et al. 2017 , to LLMs. However, this adaptation is more expensive in computation and memory than our method, requiring second-order gradient updates and saving a model for each task. As a result, we evaluate on larger models than those tried in Sinha et al. 2024 . As a different approach, Min et al. 2022 and Chen et al. 2022 explored meta-learning for LLMs using in-context learning. These works show that it is possible to fine-tune LLMs on in-context examples and achieve generalization. However, our approach does not require curation of such examples, does not place constraints on the size of the context window of a model, and is more scalable. Uncertainty representation for LLMs. Approaches to capturing uncertainty for LLMs can rely on the intrinsic representation of uncertainty in the model or focus on capturing extrinsic uncertainty about model parameters. Intrinsic approaches produce better uncertainty calibration via prompt engineering and sampling ( Gruver et al. 2023 ) or learning an external model ( Shen et al. 2024 ) . Extrinsic approaches include use fine-tuning methods to incorporate uncertainty, such as training LoRA with ensembles ( Balabanov and Linander 2024 ) , Laplace approximation ( Yang et al. 2023 ) , and variational inference ( Wang et al. 2024 ) . Our work takes the extrinsic approach but differs from existing approaches by using the meta-learning setting to achieve strong