GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, Eunhyeok Park · 2025
Read GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning by Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, Eunhyeok Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20355.txt.
Description
Yeonjoon Jung and co-authors present GraLoRA, a fix for a ceiling in how cheaply large models can be fine-tuned. Low-rank adaptation, or LoRA, trains small add-on matrices instead of the whole model. It works best at modest ranks, then stalls or overfits when the rank grows, still trailing full fine-tuning.
The authors trace the stall to tangled gradients across unrelated input channels in LoRA's single bottleneck. GraLoRA splits each weight matrix into sub-blocks, each with its own small adapter. The split untangles the gradients at almost no extra cost. Accuracy climbs closer to full fine-tuning while the training stays light.
Catalog record
| CREATOR | Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, Eunhyeok Park |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20355