BoostLoRA: Growing Effective Rank by Boosting Adapters
Raviteja Anantha, Nick Levato, Layne C. Price · 2026
Read BoostLoRA: Growing Effective Rank by Boosting Adapters by Raviteja Anantha, Nick Levato, Layne C. Price (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27308.txt.
Introduction
Anantha, Levato, and Price propose BoostLoRA, a gradient-boosting framework that grows effective rank by merging minimal adapters trained on the model's errors. Ultra-low-parameter adapters sit in fixed low-rank spaces that cap performance. A ROTATE SVD basis strategy assigns each round to a right-angle space, so total effective rank grows linearly with rounds each staying ultra-low-rank. After merging, adapters are discarded, leaving zero run-time extra cost. On Qwen2.5-3B it reaches 89.1 percent on GSM8K and about 69 percent on MATH-500.
Catalog record
| CREATOR | Raviteja Anantha, Nick Levato, Layne C. Price |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38641 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27308
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BoostLoRA: Growing Effective Rank by Boosting Adapters Raviteja Anantha, Nick Levato, Layne C. Price Abstract Parameter-efficient fine-tuning (PEFT) methods face a tradeoff between adapter size and expressivity: ultra-low-parameter adapters are confined to fixed low-rank subspaces, capping performance even with extended training. We propose BoostLoRA, a gradient-boosting framework that overcomes this limit by iteratively training and merging minimal adapters on the examples the current model gets wrong. A rotate SVD basis strategy assigns each round to an orthogonal subspace, so cumulative effective rank grows linearly with the number of rounds while each adapter remains ultra-low-rank. After merging, adapters are discarded, leaving zero inference overhead. On Qwen2.5-3B, BoostLoRA reaches 89.1% on GSM8K and 68.8% on MATH-500, surpassing both the best single-shot ultra-low-parameter adapter (TinyLoRA) and full fine-tuning; on code generation it reaches 57.2% on MBPP and 80.4% on HumanEval while full fine-tuning drops below the zero-shot baseline. We also demonstrate cross-architecture transfer on protein binding classification with ESM2-650M and cross-entropy training. BoostLoRA is, to our knowledge, the first PEFT method whose effective rank grows with training, separating per-round parameter cost from total representational capacity. 1 Introduction Parameter-efficient fine-tuning lets practitioners adapt large language models without updating billions of weights. LoRA [ 13 ] injects trainable low-rank matrices into frozen layers; follow-up work has pushed the Pareto frontier of parameters versus performance lower through shared random projections [ 16 ] , SVD-informed initialization [ 22 , 35 ] , and weight tying [ 5 ] . The lowest-parameter point on this frontier is TinyLoRA [ 24 ] , which reduces trainable parameters to as few as one parameter per module by projecting a low-dimensional vector through fixed random matrices within the SVD subspace of each weight matrix. Combined with reinforcement learning (RL) [ 30 ] , a 16-parameter TinyLoRA adapter improves Qwen2.5-3B from 76% to 81% on GSM8K, showing that RL is far more parameter-efficient than supervised fine-tuning (SFT) at this scale. However, individual ultra-low-parameter adapters hit an expressivity limit [ 33 ] : each adapter is confined to a fixed-rank subspace, and adding more parameters yields diminishing returns beyond a threshold. TinyLoRA’s best configuration (8,064 parameters) plateaus at 87.2% on GSM8K, within a point of full fine-tuning (87.0%) but unable to surpass it. We propose BoostLoRA , a PEFT framework based on gradient boosting [ 11 ] , which trains T sequential TinyLoRA adapters, each with ultra-small parameter budgets (e.g., 12 trainable parameters), on the examples the current model gets wrong. After each round the adapter is merged into the base weights and discarded, leaving no inference overhead. The core mechanism is a rotate SVD basis strategy that assigns each round’s adapter to an orthogonal subspace, so the cumulative weight update reaches effective rank \min(rT,d) where each adapter is rank r . With r=2 and T=20 rounds, BoostLoRA builds the capacity of a rank-40 update while each round optimizes over a well-conditioned 2\times 2 rotation matrix. On GSM8K, BoostLoRA reaches 89.1% , beating TinyLoRA’s best (87.2% at 8,064 parameters) and full fine-tuning (87.0% at 3.09B parameters) by 1.9 and 2.1 points. On MATH-500 it reaches 68.8%, and on code generation it achieves 57.2% on MBPP and 80.4% on HumanEval. We also test BoostLoRA outside of decoder models and generation: on PPB-Affinity [ 20 ] protein binding classification with ESM2-650M [ 19 ] and cross-entropy training, it beats single-shot TinyLoRA at every parameter scale tested. We contribute: 1. BoostLoRA , a gradient-boosted PEFT framework that resolves expressivity limits of ultra-low-parameter adapters by sequentially training and merging TinyLoRA adapters on failures (§ 3 ). 2. A rotate SVD basis strategy that ensures linear effective rank growth ( \leq rT ) across boosting rounds, with empirical confirmation that the bound is tight (§ 3.3 , § 5 ). 3. An analysis of gradient isolation from correct examples (§ 3.2 ), exact rank growth (Theorem 1 ), and a non-vacuous generalization bound (Theorem 2 ), which we validate empirically (§ 5 ). 4. Results across math, code, and protein tasks on both decoder and encoder models, with both GRPO and cross-entropy training (§ 5 ). 2 Related Work Parameter-efficient fine-tuning. LoRA [ 13 ] showed that low-rank updates can match full fine-tuning at a fraction of the parameter cost, motivating many variants. AdaLoRA [ 35 ] allocates rank budgets adaptively across layers via importance scoring. LoRA-XS [ 5 ] inserts a small r\times r trainable core between frozen SVD factors. VeRA [ 16 ] shares a single pair of frozen random matrices across layers, learning only per-layer scaling vectors. DoRA [ 21 ] decomposes weights into magnitude and direction, applying LoRA to the directional component, which stabilizes training. PiSSA [ 22 ] initializes LoRA factors with the principal singular values and vectors, training the most important weight directions directly. TinyLoRA [ 24 ] uses a trainable vector projected through fixed random matrices in the SVD subspace reduces adapters to as few as one parameter. All of these methods share the limitation that the effective rank of the weight update is fixed at adapter creation and cannot grow during or after training. To our knowledge, BoostLoRA is the first PEFT method where cumulative effective rank grows with training. Iterative and boosted adaptation. Several works have explored iterative training and merging of adapters. COLA [ 32 ] applies Frank-Wolfe optimization to iteratively train and merge standard-rank LoRA modules. However, COLA does not perform residual fitting (each round trains on the full dataset, not the failure set) and uses standard-rank LoRA rather than ultra-low-parameter adapters. MELoRA [ 27 ] stacks mini-LoRA modules in a block-diagonal arrangement to increase effective rank without additional parameters, but within a single training round. Model Soups [ 31 ] and LoRA Soups [ 25 ] explore weight averaging of independently trained adapters for skill composition, but do not perform sequential residual fitting. ReLoRA [ 17 ] periodically merges and restarts standard-rank LoRA during pre-training, achieving high-rank updates through accumulation. However, ReLoRA uses millions of parameters per round, trains on the full dataset (no failure focusing), and does not enforce orthogonality between rounds. XGBLoRA [ 36 ] proposed the closest idea to ours: gradient boosting with rank-1 LoRA adapters. However, XGBLoRA differs in three key ways: (i) its “rank-1” adapters still have {\sim} 8,192 parameters per layer ( 683\times our 12 total); (ii) it uses SFT, which TinyLoRA [ 24 ] showed is 100–1,000 times less parameter-efficient than RL at this scale; and (iii) it was evaluated only on commonsense reasoning (BoolQ, PIQA), not math or code. RL for reasoning. GRPO [ 30 ] eliminates the critic network of PPO [ 29 ] by estimating advantages from within-group reward statistics. DeepSeek-R1 [ 12 ] showed that reasoning behaviors arise from pure GRPO training at scale. SimpleRL [ 34 ] investigated zero-RL training (RL directly from base models without SFT) across several model families. RLOO [ 1 ] revisited REINFORCE-style optimization for RLHF, showing that simpler algorithms match or exceed PPO. Ultra-low-parameter adapters face an expressivity limit under RL, where methods like VeRA and rank-1 LoRA fail to learn reasoning, and SVD-based initializations (PiSSA) suffer spectral collapse under RL dynamics [ 33 ] . In BoostLoRA, each adapter operates at the expressivity boundary, with the boosting