ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints

Debasmit Das, Hyoungwoo Park, Munawar Hayat, Seokeon Choi, Sungrack Yun, Fatih Porikli · 2025

Read ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints by Debasmit Das, Hyoungwoo Park, Munawar Hayat, Seokeon Choi, Sungrack Yun, Fatih Porikli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08044.txt.

Description

Debasmit Das and co-authors give low-rank adapters a smart head start instead of a random one in this 2025 paper. Adapters fine-tune giant foundation models cheaply, but they begin from random weights with one fixed size everywhere. A better birth could speed learning and lift the final score.

Their method treats the starting weights as a puzzle set by the gap between pretraining and the new task. Constraints tying the two stages' activity patterns can be rearranged into a closed formula for the weights. The formula needs only the base model and a pass of task data, with no training at all. Splitting that estimate into the adapter's two matrices also lets each spot pick its own rank. Across image generation, classification, and understanding tasks, the warm start beat random and rival data-driven starts, and side tests mapped which choices carry the gain.

Catalog record

CREATORDebasmit Das, Hyoungwoo Park, Munawar Hayat, Seokeon Choi, Sungrack Yun, Fatih Porikli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08044

Browse the full library · About the Tower of Babel