Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-task Learning

Ziyu Zhao, Yixiao Zhou, Zhi Zhang, Didi Zhu, Tao Shen, Zexi Li, Jinluan Yang, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu Yu Cheng · 2025

Read Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-task Learning by Ziyu Zhao, Yixiao Zhou, Zhi Zhang, Didi Zhu, Tao Shen, Zexi Li, Jinluan Yang, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu Yu Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15103.txt.

Description

Ziyu Zhao and co-authors refine a cheap way to adapt language models in a 2025 paper. The method, LoRA, adds small trainable matrices to a frozen model. One adapter struggles when a model must juggle many tasks at once, since tasks pull in different directions. A popular fix uses several adapters as experts and routes each input to a few of them.

The authors show that routing among many adapters equals slicing one adapter by rank and switching slices on and off. Finer slices, under the same budget, gave better results across mixed tasks. Their method, SMoRA, treats each single rank as an expert and activates ranks one by one as needed. Knowledge shared across tasks is no longer trapped inside lone adapters.

Catalog record

CREATORZiyu Zhao, Yixiao Zhou, Zhi Zhang, Didi Zhu, Tao Shen, Zexi Li, Jinluan Yang, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu Yu Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-15103

Browse the full library · About the Tower of Babel