Ensembles of Low-Rank Expert Adapters

Yinghao Li, Vianne Gao, Chao Zhang, MohamadAli Torkamani · 2025

Read Ensembles of Low-Rank Expert Adapters by Yinghao Li, Vianne Gao, Chao Zhang, MohamadAli Torkamani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00089.txt.

Introduction

Yinghao Li and co-authors split language model fine-tuning into a team of small specialists. Their 2025 paper notes that mixed training data pulls a model in clashing directions, which blurs skill across tasks. Their framework, ELREA, first groups training examples by the direction their gradients push, so each cluster shares one area of expertise.

A small low-rank adapter then trains on each cluster alone, staying cheap and modular. At answer time, the system checks which clusters the input resembles and blends the matching adapters' predictions. Across domain tasks, the ensemble beats a single adapter trained on everything, and beats other ensemble designs at similar cost. The authors frame the split as a way to keep skills sharp instead of averaged away.

Catalog record

CREATORYinghao Li, Vianne Gao, Chao Zhang, MohamadAli Torkamani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-00089

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ensembles of Low-Rank Expert Adapters Yinghao Li, Vianne Gao, Chao Zhang, MohamadAli Torkamani Abstract The training and fine-tuning of large language models (LLMs) often involve diverse textual data from multiple sources, which poses challenges due to conflicting gradient directions, hindering optimization and specialization. These challenges can undermine model generalization across tasks, resulting in reduced downstream performance. Recent research suggests that fine-tuning LLMs on carefully selected, task-specific subsets of data can match or even surpass the performance of using the entire dataset. Building on these insights, we propose the Ensembles of Low-Rank Expert Adapters ( ELREA ) framework to improve the model’s capability to handle diverse tasks. ELREA clusters the training instructions based on their gradient directions, representing different areas of expertise and thereby reducing conflicts during optimization. Expert adapters are then trained on these clusters, utilizing the low-rank adaptation (LoRA) technique to ensure training efficiency and model scalability. During inference, ELREA combines predictions from the most relevant expert adapters based on the input data’s gradient similarity to the training clusters, ensuring optimal adapter selection for each task. Experiments show that our method outperforms baseline LoRA adapters trained on the full dataset and other ensemble approaches with similar training and inference complexity across a range of domain-specific tasks. 1 Introduction While general-domain large language models (LLMs) such as GPT-4 ( OpenAI, 2022 ; OpenAI, 2023 ) and Llama ( Touvron et al., 2023 ) have shown remarkable efficacy in diverse applications, adapting these models through supervised fine-tuning to specific domains or tasks remains indispensable for achieving optimal performance. For example, instruction following requires subtle model adjustments to specialized datasets that the general pre-training corpus alone cannot provide ( Ouyang et al., 2022 ) . Significant resources have been invested in constructing varied, high-quality datasets tailored for LLM fine-tuning such as Alpaca ( Taori et al., 2023 ) , the Pile ( Gao et al., 2021 ) , or Flan ( Longpre et al., 2023 ) . These efforts have fueled the development of specialized models that address complex tasks across fields such as medical diagnostics ( Singhal et al., 2023 ) , financial analytics ( Yang et al., 2023 ) , and scientific decision-making ( Zhang et al., 2024b ) , or to provide reasoning to their results ( Wei et al., 2022 ) , which were tasks once deemed challenging for automated systems. Nonetheless, fine-tuning LLMs on a comprehensive dataset frequently encounters the issue of conflicting gradient directions from varied training data points ( Wang et al., 2021 ; Xia et al., 2024 ; Chen et al., 2024 ) . This phenomenon complicates the update process of models, potentially leading to suboptimal performance. Wang et al. (2023d) demonstrate that mixing diverse instructional datasets can sometimes result in less than ideal outcomes compared to fine-tuning on a carefully selected subset of the data that directly addresses the task at hand. To enhance the relevance of training data to specific tasks, Xie et al. (2023) have proposed methods like importance resampling, which aligns the training dataset more closely with the target task distribution. Another innovative approach proposed by Xia et al. (2024) , termed targeted instruction tuning, involves selecting a small percentage (about 5%) of training data that most significantly influences task performance based on the average gradients of tokens. This method has shown promise, achieving comparable or superior results to traditional full dataset fine-tuning across various tasks. In addition, Xia et al. (2024) also present better outcomes in selecting data points based on the gradient norm than sentence embeddings. Despite these advancements, current data selection techniques for fine-tuning are predominantly target-driven, relying heavily on specific features of the target task ( e.g. , n-gram frequency, example answer embedding, gradient direction) to guide the selection process. This requirement for task-specific data features imposes significant limitations when adapting LLMs to new or emerging tasks, especially when relevant training data or features are unavailable. To address these challenges, we propose a novel framework, Ensembles of Low-Rank Expert Adapters ( ELREA ), which leverages Low-Rank Adaptation (LoRA; Hu et al., 2022 ; Dettmers et al., 2023 ) to create multiple expert adapters. These adapters are trained independently on data groups with similar gradient directions and their predictions are assembled during inference based on the gradient features of the input. Specifically, ELREA begins by fine-tuning a base adapter on the full dataset to capture a wide range of general knowledge. We then evaluate and cluster the gradients of individual data points relative to their influence on the base adapter, organizing them into similarly sized groups. On each cluster we continue training a specialized LoRA expert that is initialized from the base adapter, allowing the training process to maintain a comparable computational burden to that of a single adapter trained on the entire dataset. During inference, the expert adapters collaboratively determine the output by dynamically weighting the adapters according to their alignment with the clusters’ gradient profile. Compared with conventional Deep Ensembles, such calculation could be conducted only once at the beginning in the recurrent generation process and re-used in subsequent passes, causing minimal computational overhead while achieving stronger performance ( Lakshminarayanan et al., 2017 ; Havasi et al., 2021 ; Wang et al., 2023a ) . Unlike previous methods, ELREA focuses on the task-agnostic setup, i.e. , a one-time training effort without the need for additional task-specific validation data, making it more suitable for real-world deployment of LLMs. In summary, our contributions are threefold: • We introduce Ensembles of Low-Rank Expert Adapters ( ELREA ), a framework that integrates efficient parameter adaptation techniques into an ensemble model to address conflicting gradient directions in LLM fine-tuning. • By combining gradient features with clustering methods, we create expert adapters specialized for different gradient profiles, enabling the model to adapt to diverse tasks without relying on task-specific data features or validation data points. • We demonstrate that ELREA outperforms baseline LoRA adapters trained on the full dataset across various domain-specific applications, as well as other Mixture of Experts (MoE) and self-consistency methods. 2 Preliminaries 2.1 Language Models and Parameter-Efficient Fine-Tuning Decoder-only LMs, pioneered by GPT ( Radford et al., 2018 ) , are built upon the decoder component of the Transformer architecture ( Vaswani et al., 2017 ) and are among the most prevalent and thoroughly examined language models today. A pre-trained LM, denoted as {\mathcal{M}} , learns the language patterns on extensive text corpora {\mathcal{D}}_{\text{pre-train}} through an unsupervised next-token-prediction (NTP) objective, which minimized the negative log likelihood (NLL) of a subsequent token x_{t} in a length- T sequence \bm{x}\in{\mathcal{V}}^{T} consisting tokens from the vocabulary {\mathcal{V}} based on the preceding context \bm{x}_{<t} : {\mathcal{L}}_{\text{NTP}}(\bm{x})=-\sum_{t=1}^{T}\log P(x_{t}|\bm{x}_{<t};\bm{\theta}_{{\mathcal{M}}}), (1) where \bm{\theta}_{{\mathcal{M}}} are the network parameters of the LM. Originally designed for text completion, the pretrained LMs have been enhanced with instruction-following or task-specific capabilities through targeted fine-tuning ( Ouyang et al., 2022 ; OpenAI, 2022 ; OpenAI, 2023 ) , expanding their utility across diverse applications. The fine-tuning process frequently adopts the NTP objective, utilizing a smaller, specialized fine-tuning dataset {\mathcal{D}}_{\text{ft}} that consists of instruction-response pairs \bm{x}_{\text{ft}}=(\bm{x}_{\text{instr}},\bm{x}_{\text{resp}}) . Full-parameter fine-tuning of high-performing LMs, which involves calculating \nabla_{\bm{\theta}_{{\mathcal{M}}}}\mathcal{L}_{\text{NTP}}(\bm{x}) and updating \bm{\theta}_{\mathcal{M}} accordingly, is often impractical due to computational constraints arising from their vast

Read the complete text