AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
Ziyun Liu Fengmiao Bian Jian-Feng Cai Department of MathematicsThe Hong Kong University of Science and Technology [email protected] {mafmbian,jfcai}@ust.hk · 2026
Read AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation by Ziyun Liu Fengmiao Bian Jian-Feng Cai Department of MathematicsThe Hong Kong University of Science and Technology [email protected] {mafmbian,jfcai}@ust.hk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08734.txt.
Description
Ziyun Liu, Fengmiao Bian, and Jian-Feng Cai rethink the optimizers used to train low rank adapters, the small trainable patches popular for fine tuning large models. The adapter's factor geometry makes the natural preconditioning matrix singular, so a preconditioned direction cannot be uniquely pulled back into factor updates. The authors map existing optimizers into four families based on how they work around that singularity.
Their AdaPreLoRA pairs a memory light diagonal preconditioner with a closed form factor solve that best balances the two factors' contributions. Across language model benchmarks and diffusion personalization, it matches or beats representative rivals while keeping memory at normal adapter levels. Fine tuning practice gains a principled optimizer that respects adapter geometry.
Catalog record
| CREATOR | Ziyun Liu Fengmiao Bian Jian-Feng Cai Department of MathematicsThe Hong Kong University of Science and Technology [email protected] {mafmbian,jfcai}@ust.hk |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08734