FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation

Muqing Liu, Chongjie Si, Yuheng Jia, AI Institute, School of Computer Science, Shanghai Jiao Tong University School of, Computer Science, Engineering, Southeast University Key Laboratory of New GenerationArtificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University) · 2026

Read FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation by Muqing Liu, Chongjie Si, Yuheng Jia, AI Institute, School of Computer Science, Shanghai Jiao Tong University School of, Computer Science, Engineering, Southeast University Key Laboratory of New GenerationArtificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22905.txt.

Description

Muqing Liu, Chongjie Si, and Yuheng Jia make a popular fine-tuning method more flexible. Full fine-tuning of large models costs too much memory and compute for many teams. Low-Rank Adaptation, or LoRA, trains small added matrices instead, but it fixes their rank in advance.

A fixed rank wastes capacity on some layers and starves others. FlexLoRA uses entropy, a measure of spread, to guide where rank should grow or shrink. Flexible ranks could give small teams big-model quality at a lower training bill. Entropy guides the flexibility, giving layers with richer signals more rank while trimming layers that need less capacity.

Catalog record

CREATORMuqing Liu, Chongjie Si, Yuheng Jia, AI Institute, School of Computer Science, Shanghai Jiao Tong University School of, Computer Science, Engineering, Southeast University Key Laboratory of New GenerationArtificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University)
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22905

Browse the full library · About the Tower of Babel