UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter Efficient Fine-Tuning of Large Models

Xueyan Zhang, Jinman Zhao, Zhifei Yang, Yibo Zhong, Shuhao Guan, Linbo Cao , Yining Wang · 2025

Read UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter Efficient Fine-Tuning of Large Models by Xueyan Zhang, Jinman Zhao, Zhifei Yang, Yibo Zhong, Shuhao Guan, Linbo Cao , Yining Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20154.txt.

Description

Xueyan Zhang and co-authors fine tune large language models by selectively resetting rows and columns of frozen weight matrices, guided by vector size, in work from twenty twenty-five. Full fine tuning is costly, so lean adapters train a small slice instead. Popular low rank methods still carry more trainable weights than needed.

Their method, UORA, reparametrizes the frozen matrices through interpolation and refreshes chosen lines in an even, orthogonal fashion. The trainable count drops well below the leading low rank baseline, with better compute and storage economy than a shared vector rival. Benchmarks on language understanding, generation, instruction tuning, and image classification held accuracy with negligible overhead.

Catalog record

CREATORXueyan Zhang, Jinman Zhao, Zhifei Yang, Yibo Zhong, Shuhao Guan, Linbo Cao , Yining Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20154

Browse the full library · About the Tower of Babel