ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints
Debasmit Das, Hyoungwoo Park, Munawar Hayat, Seokeon Choi, Sungrack Yun, Fatih Porikli · 2025
Read ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints by Debasmit Das, Hyoungwoo Park, Munawar Hayat, Seokeon Choi, Sungrack Yun, Fatih Porikli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08044.txt.
Description
Debasmit Das and co-authors give low-rank adapters a smart head start instead of a random one in this 2025 paper. Adapters fine-tune giant foundation models cheaply, but they begin from random weights with one fixed size everywhere. A better birth could speed learning and lift the final score.
Their method treats the starting weights as a puzzle set by the gap between pretraining and the new task. Constraints tying the two stages' activity patterns can be rearranged into a closed formula for the weights. The formula needs only the base model and a pass of task data, with no training at all. Splitting that estimate into the adapter's two matrices also lets each spot pick its own rank. Across image generation, classification, and understanding tasks, the warm start beat random and rival data-driven starts, and side tests mapped which choices carry the gain.
Catalog record
| CREATOR | Debasmit Das, Hyoungwoo Park, Munawar Hayat, Seokeon Choi, Sungrack Yun, Fatih Porikli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08044