ALTO : Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads

Jingwei Zuo, Xinze Feng, Zien Liu, Kaijian Wang, Fanjiang Ye, Ye Cao, Zhuang Wang, Yuke Wang, [1ex] Rice University · 2026

Read ALTO : Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads by Jingwei Zuo, Xinze Feng, Zien Liu, Kaijian Wang, Fanjiang Ye, Ye Cao, Zhuang Wang, Yuke Wang, [1ex] Rice University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05426.txt.

Introduction

Zuo and seven colleagues speed up tuning of LoRA adapters, the small add-on modules that adapt large language models, where many parallel jobs waste compute on weak settings and idle graphics chips. Their ALTO system kills weak configs early from loss trends, packs surviving adapters together with grouped matrix math, and schedules jobs across tasks. Tests report speedups up to 13.8 times over leading systems with no loss in adapter quality.

Catalog record

CREATORJingwei Zuo, Xinze Feng, Zien Liu, Kaijian Wang, Fanjiang Ye, Ye Cao, Zhuang Wang, Yuke Wang, [1ex] Rice University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66951
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05426

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ALTO : Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads Jingwei Zuo, Xinze Feng, Zien Liu, Kaijian Wang, Fanjiang Ye, Ye Cao, Zhuang Wang, Yuke Wang, [1ex] Rice University Abstract Low-Rank Adaptation (LoRA) is now the dominant method for parameter-efficient fine-tuning of large language models, but achieving a high-quality adapter often requires systematic hyperparameter tuning because LoRA performance is highly sensitive to configuration choices. In practice, this leads to many concurrent LoRA jobs, often spanning heterogeneous tasks in multi-tenant environments. Existing systems largely handle these jobs independently, which both wastes computation on weak candidates and leaves GPUs underutilized. We present ALTO ( A daptive L oRA T uning and O rchestration), a co-designed training system that accelerates LoRA hyperparameter tuning while enabling efficient cluster sharing across heterogeneous tasks. The central insight behind ALTO is that when multiple tuning jobs run concurrently over a shared frozen backbone, they expose optimization opportunities that single-job designs cannot exploit. Building on this, ALTO monitors loss trajectories to terminate unpromising configurations early, uses fused grouped GEMM together with a new rank-local adapter parallelism to co-locate surviving adapters and reclaim freed GPU capacity, and combines intra-task and inter-task scheduling to improve multi-task placement by leveraging the predictable duration of LoRA jobs. Extensive evaluation shows that ALTO achieves up to 13.8 × 13.8\times speedup over state-of-the-art without sacrificing adapter quality. † † ∗ Jingwei Zuo and Xinze Feng contributed equally. 1 Introduction Pre-trained Large Language Models (LLMs), such as Claude 3 [ 5 ] and GPT-4 [ 2 ] , have shown strong performance across a broad range of tasks, including coding [ 9 , 27 ] , multilingual understanding [ 54 ] , and conversational assistance [ 66 , 10 ] . The paradigm of pre-training followed by fine-tuning has enabled these models to further achieve state-of-the-art performance when adapted to personalized or domain-specific applications, such as legal analysis [ 21 ] , personalized education [ 7 ] , and customer support [ 61 ] . However, fine-tuning large models with all model parameters updated poses a significant challenge due to the high computational cost of training and serving numerous fine-tuned variants. To address this, parameter-efficient fine-tuning (PEFT) techniques have emerged as scalable alternatives to traditional full fine-tuning, among which Low-Rank Adaptation (LoRA) [ 25 ] is particularly popular due to its simplicity and effectiveness. LoRA significantly reduces the number of trainable parameters by introducing low-rank decomposition matrices into model layers, allowing for specialization while keeping the pre-trained model weights frozen. Figure 1 : Different hyperparameters yield significantly varying downstream task performance in LoRA training. (a) Best validation loss distributions across 165 hyperparameter configs (varying batch size, learning rate, and LoRA rank) for Llama-3.1-8B. (b) GSM8K test accuracy of the best checkpoint per config shows a maximum of 73.9% difference. (c) DPO Reward accuracy of 60 different configurations on UltraFeedback dataset shows a maximum of 26.7% difference. Figure 2 : System Overview of ALTO . Due to these substantial resource savings, LoRA fine-tuning has been widely adopted in both large-scale cloud platforms [ 17 , 58 , 56 ] and resource-constrained local environments [ 13 , 67 ] for LLM post-training methods, including supervised fine-tuning (SFT) [ 44 ] and reinforcement learning (RL) [ 52 ] . Specifically, a LoRA fine-tuning task takes as input a base model and a training dataset, and aims to produce a high-quality LoRA adapter, as measured by task-specific metrics such as accuracy in SFT or reward in RL. Similarly to model pre-training, LoRA fine-tuning also requires specifying hyperparameters, such as learning rate, batch size, and adapter rank. We denote a LoRA fine-tuning job as the training of a LoRA adapter under a specific hyperparameter configuration. A LoRA task may consist of one or more fine-tuning jobs. Although a LoRA fine-tuning job offers substantial efficiency gains over full parameter fine-tuning, how to efficiently train a LoRA task to obtain a decent LoRA adapter is challenging. Our large-scale empirical study reveals the necessity of hyperparameter search in a LoRA task, and there is no single rule of thumb for hyperparameter tuning (§ 3 ). We conducted extensive experiments on hyperparameter sensitivity in LoRA fine-tuning across various model architectures, model sizes, and downstream tasks in both SFT and RL. As shown in Figure 1 , the adapter quality gap between the best and worst configurations can be up to 73.9%. Unfortunately, existing LoRA fine-tuning frameworks, such as PEFT [ 40 ] , LLamaFactory [ 67 ] or Unsloth [ 13 ] , are unaware of this hyperparameter sensitivity and instead treat the LoRA task as a passive and sequential execution of provided jobs. Consequently, practitioners have to manually launch a sequence of LoRA fine-tuning jobs over a hyperparameter search space to obtain a high-quality LoRA adapter for a given task, making the task-complete time increase linearly with the search space. Additionally, because interests and goals often differ between tasks, this pattern of multiple jobs naturally extends to multiple tasks submitted by a single user or many users in shared training environments such as cloud platforms [ 38 , 64 , 18 , 3 , 41 ] . To support these multi-job and multi-task workloads, existing systems [ 40 , 67 , 13 ] typically execute LoRA jobs separately, often by training one adapter at a time and switching the active adapter between jobs. Because individual LoRA jobs are lightweight, this approach results in low hardware utilization and, when scaled across multiple GPUs, disproportionately high distributed overhead. A common optimization strategy is multi-LoRA fine-tuning [ 64 , 62 , 23 , 33 ] , which runs multiple LoRA jobs jointly on the same GPUs. However, they still operate at job granularity and do not account for heterogeneity across tasks and job progress, leaving substantial GPU resources underutilized in multi-task workloads. To explore optimization opportunities to address these inefficiencies, we identify several key observations. First , LoRA hyperparameter quality is highly uneven across candidates over the course of training and does not reliably transfer between tasks. Within each LoRA task, many underperforming jobs can be identified early using objective-aligned training metrics, such as the supervised loss in SFT, because these signals directly measure progress toward the task objective [ 46 ] . This motivates a dynamic, performance-aware system design that improves training efficiency while ensuring strong downstream LoRA performance. Second , we observe a fundamental conflict between the statistical behavior and system efficiency of LoRA fine-tuning. Specifically, our analysis (§ 3 ) suggests that small batch sizes are statistically preferred for strong LoRA convergence, whereas sufficiently large batch sizes are preferred in practice for high hardware utilization. Moreover, in multi-GPU settings, common distributed strategies either do not support batch sizes smaller than the world size [ 65 , 49 ] or suffer from pipeline bubbles [ 16 , 26 , 42 ] and frequent communication overhead [ 55 , 43 ] . Addressing this mismatch requires a coordinated approach that jointly optimizes convergence quality and hardware efficiency. Third , although realistic LoRA workloads are heterogeneous along many dimensions, such as training dataset size, their execution durations can often be estimated reliably before training begins. This predictability creates optimization opportunities for workload scheduling to

Read the complete text