GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, Eunhyeok Park · 2025

Read GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning by Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, Eunhyeok Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20355.txt.

Description

Yeonjoon Jung and co-authors present GraLoRA, a fix for a ceiling in how cheaply large models can be fine-tuned. Low-rank adaptation, or LoRA, trains small add-on matrices instead of the whole model. It works best at modest ranks, then stalls or overfits when the rank grows, still trailing full fine-tuning.

The authors trace the stall to tangled gradients across unrelated input channels in LoRA's single bottleneck. GraLoRA splits each weight matrix into sub-blocks, each with its own small adapter. The split untangles the gradients at almost no extra cost. Accuracy climbs closer to full fine-tuning while the training stays light.

Catalog record

CREATORYeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, Eunhyeok Park
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20355

Browse the full library · About the Tower of Babel