AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation

Ziyun Liu Fengmiao Bian Jian-Feng Cai Department of MathematicsThe Hong Kong University of Science and Technology [email protected] {mafmbian,jfcai}@ust.hk · 2026

Read AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation by Ziyun Liu Fengmiao Bian Jian-Feng Cai Department of MathematicsThe Hong Kong University of Science and Technology [email protected] {mafmbian,jfcai}@ust.hk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08734.txt.

Description

Ziyun Liu, Fengmiao Bian, and Jian-Feng Cai rethink the optimizers used to train low rank adapters, the small trainable patches popular for fine tuning large models. The adapter's factor geometry makes the natural preconditioning matrix singular, so a preconditioned direction cannot be uniquely pulled back into factor updates. The authors map existing optimizers into four families based on how they work around that singularity.

Their AdaPreLoRA pairs a memory light diagonal preconditioner with a closed form factor solve that best balances the two factors' contributions. Across language model benchmarks and diffusion personalization, it matches or beats representative rivals while keeping memory at normal adapter levels. Fine tuning practice gains a principled optimizer that respects adapter geometry.

Catalog record

CREATORZiyun Liu Fengmiao Bian Jian-Feng Cai Department of MathematicsThe Hong Kong University of Science and Technology [email protected] {mafmbian,jfcai}@ust.hk
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08734

Browse the full library · About the Tower of Babel