Supplement to “RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts”

Tuan Truong, Chau Nguyen, Huy Nguyen, Minh Le, Trung Le, Nhat Ho · 2025

Read Supplement to “RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts” by Tuan Truong, Chau Nguyen, Huy Nguyen, Minh Le, Trung Le, Nhat Ho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03044.txt.

Description

Tuan Truong and co-authors rethink LoRA, a popular cheap way to fine-tune giant AI models by training only small low-rank add-on matrices. Why LoRA works so well lacked theory. Viewing it through mixture-of-experts models, the team shows that reparameterizing those small matrices can speed up learning dramatically.

In their analysis, the right reparameterization cuts the data needed for a target accuracy from an exponential amount to a polynomial one. Their method, RepLoRA, inserts tiny neural networks to reshape the LoRA matrices during training. Across several domains it consistently beat plain LoRA. With scarce data the margin reached forty percent, and RepLoRA matched LoRA's full-data performance using only thirty percent of the data. The results give LoRA a firmer theoretical footing and a practical upgrade.

Catalog record

CREATORTuan Truong, Chau Nguyen, Huy Nguyen, Minh Le, Trung Le, Nhat Ho
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE140 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-03044

Browse the full library · About the Tower of Babel