AdaPaD : Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery

Barbara Su, Fangshuo Liao, Anastasios Kyrillidis · 2026

Read AdaPaD : Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery by Barbara Su, Fangshuo Liao, Anastasios Kyrillidis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10741.txt.

Introduction

Barbara Su, Fangshuo Liao, and Anastasios Kyrillidis introduce AdaPaD, a 2026 approach to fine-tuning large language models that removes the need to fix a rank in advance. In LoRA fine-tuning, rank is a number that sets the size of the update a model may learn, and picking it badly wastes effort or caps what the model can learn.

Instead of pulling out one component at a time, AdaPaD trains all of them side by side, letting each one correct itself as its predecessors improve. It also grows the rank where the data shows it matters, stopping when a shared budget runs out. Tests on standard language tasks match strong rivals, and the authors prove each component's error shrinks fast after a warm-up period.

Catalog record

CREATORBarbara Su, Fangshuo Liao, Anastasios Kyrillidis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97142
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10741

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AdaPaD : Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery Barbara Su, Fangshuo Liao, Anastasios Kyrillidis Abstract Fine-tuning large language models with LoRA ( 17 ) requires choosing a rank r r before training starts. Existing approaches either extract rank-1 components sequentially ( 29 ; 48 ) , freezing each component’s error permanently into every subsequent residual, or optimize the full low-rank factorization jointly ( 52 ; 43 ) with guarantees that describe only the joint update, not individual rank-1 directions. We present AdaPaD (Adaptive Parallel Deflation), which trains all rank-1 components simultaneously: each worker refines its component against a deflation target built from the latest estimates of all predecessors, and as those estimates improve, the targets improve too. We call this property self-correction : deflation errors converge to zero over rounds rather than persisting as fixed residuals. On top of this backbone, AdaPaD adds advance learning (private pre-training before activation) and per-module dynamic rank discovery (importance-based growth until a shared budget is exhausted), making the rank distribution an output rather than an input. We prove that every component’s error decays exponentially after a warm-up period, with a generalization bound that splits into a vanishing algorithmic term and an irreducible statistical floor. Empirically, AdaPaD is competitive with adaptive-rank LoRA baselines on GLUE with DeBERTaV3-base at matched parameter budgets, and competitive with fixed-rank LoRA on Qwen3-0.6B SQuAD/SQuAD v2 while deploying an adapter that is on average 30.7 % 30.7\% smaller. 1 Introduction Parameter-efficient fine-tuning (PEFT) is the standard way to adapt large pretrained models to downstream tasks. Among the many approaches ( 15 ; 27 ; 2 ) , Low-Rank Adaptation (LoRA) ( 17 ) stands out for its simplicity: for a pretrained weight \mathbf{W}\in\mathbb{R}^{m\times d} , LoRA freezes \mathbf{W} and trains a low-rank update \Delta\mathbf{W}=\mathbf{B}\mathbf{A}^{\top} with \mathbf{B}\in\mathbb{R}^{m\times r} , \mathbf{A}\in\mathbb{R}^{d\times r} and rank r\ll\min(m,d) a hyperparameter. LoRA is effective in practice, but leaves a fundamental design choice unresolved: how should r be chosen? In practice r is set by hand and applied uniformly to every layer ( 17 ; 12 ; 31 ) , yet the appropriate rank varies across modules, tasks, and training stages. Adaptive-rank methods address this in different ways: AdaLoRA ( 54 ) prunes from high rank using SVD importance, IncreLoRA ( 53 ) grows from low rank by activating reserve components, DyLoRA ( 47 ) trains across a range of ranks, and SoRA ( 6 ) learns sparse gates. Recent theoretical work analyzes the joint \mathbf{B}\mathbf{A}^{\top} update: dEBORA ( 52 ) uses bilevel Frank-Wolfe with rank-identifiability guarantees, and ARD-LoRA ( 43 ) adds learnable per-head scaling with a stationary-point theorem. Both treat the full low-rank matrix as one object; we instead decompose the problem into independent rank-1 subproblems trained in parallel and analyze each separately, yielding a guarantee joint optimization cannot express: self-correction , where the deflation mismatch in each rank-1 worker provably vanishes over rounds, so individual rank-1 directions can be certified correct rather than only their joint sum. Key insight: parallel deflation with self-correction. In sequential deflation, errors in early components are permanently frozen and compound multiplicatively through the spectral gap ( 48 ) . We remove this dependency by training all rank-1 components simultaneously : each is assigned to a dedicated worker that, at every communication round, rebuilds its deflation target from the latest predecessor estimates, so early mistakes are progressively corrected rather than locked in. We call this property self-correction and prove that the deflation mismatch \|\mathbf{Y}_{k,\ell}-\mathbf{Y}_{k}^{\star}\|_{F} vanishes as \ell grows ( Section 4 ), in contrast to sequential deflation ( 48 ; 29 ) where the mismatch is fixed and generically nonzero. The bilinear setting differs from symmetric eigendecomposition ( 30 ) : the asymmetric structure \mathbf{b}\mathbf{a}^{\top}\mathbf{X} calls for Wedin perturbation theory ( 50 ) and a data-dependent recovery scale. Two further mechanisms make parallel deflation practical for LoRA: advance learning (private warm-up of dormant components against improving targets) and per-module dynamic rank discovery (importance-driven growth under a global budget). Contributions. C1. AdaPaD algorithm ( Algorithms 1 and 2 ). Parallel rank-1 deflation with advance learning (private pre-training before activation) and per-module dynamic rank discovery (importance-based growth under a shared budget); the rank distribution is an output, not an input. C2. Theoretical guarantees. Exponential convergence with vanishing deflation mismatch ( Theorems 1 and 4 ) and a weight-recovery bound that splits into a vanishing algorithmic term plus an irreducible statistical floor ( Theorems 3 and 4 ); to our knowledge, the first self-correcting guarantee for parallel rank-1 deflation in bilinear regression, structurally distinct from the joint-update guarantees of ( 52 ; 43 ) . C3. Empirical evaluation. At approximately matched budgets ( 0.32 – 0.47 M) on GLUE with DeBERTaV3-base, AdaPaD is competitive with IncreLoRA, AdaLoRA, SoRA, and dEBORA across all eight tasks ( Section 5.1 ); on Qwen3-0.6B SQuAD/SQuAD v2 it matches fixed-rank LoRA on F1/EM with adapters 30.7\% smaller on average ( Section 5.2 ); component-parallel sharding delivers up to 2.66{\times} speedup on 4{\times} H200 ( Section 5.3 ). Related work. Deflation-based methods. Hotelling’s deflation ( 14 ) is the classical approach for extracting principal components; 32 extended it to sparse PCA. Error propagation in inexact sequential deflation was analyzed by 29 and 48 . The model-parallel perspective was introduced by EigenGame ( 9 ; 10 ) , which casts top- K PCA as a multi-player game. 30 gave the first provable parallel deflation algorithm for PCA; we develop parallel deflation for the bilinear regression setting, where the asymmetric structure \mathbf{b}\mathbf{a}^{\top}\mathbf{X} requires Wedin-type perturbation analysis and the self-correction guarantee is tied to a data-dependent recovery scale. ReLoRA ( 28 ) trains a sequence of low-rank updates that sum to a high-rank result, which is sequential deflation under a different name; our approach differs by running all components in parallel with provable self-correction. Low-rank optimization. Problem ( 1 ) has been attacked via nuclear norm minimization ( 5 ; 41 ) , projected gradient descent ( 19 ; 23 ) , and factored methods ( 4 ; 46 ; 24 ; 36 ; 3 ; 35 ; 21 ; 11 ; 16 ) . These optimize the full low-rank matrix jointly but do not decompose into independent rank-1 subproblems. 22 show that fixed-rank LoRA training converges to a low-rank global minimum or diverges detectably, but this addresses the fixed-rank setting without rank adaptation. Adaptive-rank methods with convergence theory. dEBORA ( 52 ) formulates rank selection as bilevel optimization: an upper-level Frank-Wolfe on an \ell_{1} -constrained simplex selects singular-value scaling factors while a lower-level solver trains the basis matrices. dEBORA proves \mathcal{O}(1/\sqrt{T}) convergence of the upper level and, notably, finite-time rank identification (the algorithm provably discovers which singular values are zero). As a joint optimization method, dEBORA does not decompose into rank-1 subproblems and therefore does not face the deflation error propagation problem; its convergence guarantee applies to the upper-level simplex variable, not to the bilinear \mathbf{B}\mathbf{A}^{\top} factorization itself. ARD-LoRA ( 43 ) introduces learnable per-head scaling factors with \mathcal{O}(1/T) convergence to a stationary point via a standard descent lemma; the guarantee is generic smooth optimization and does not exploit the bilinear structure. Neither decomposes into rank-1 subproblems nor exhibits self-correction ( Section 4 ). 2 Background and Problem Setup

Read the complete text