Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-task Learning
Ziyu Zhao, Yixiao Zhou, Zhi Zhang, Didi Zhu, Tao Shen, Zexi Li, Jinluan Yang, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu Yu Cheng · 2025
Read Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-task Learning by Ziyu Zhao, Yixiao Zhou, Zhi Zhang, Didi Zhu, Tao Shen, Zexi Li, Jinluan Yang, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu Yu Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15103.txt.
Description
Ziyu Zhao and co-authors refine a cheap way to adapt language models in a 2025 paper. The method, LoRA, adds small trainable matrices to a frozen model. One adapter struggles when a model must juggle many tasks at once, since tasks pull in different directions. A popular fix uses several adapters as experts and routes each input to a few of them.
The authors show that routing among many adapters equals slicing one adapter by rank and switching slices on and off. Finer slices, under the same budget, gave better results across mixed tasks. Their method, SMoRA, treats each single rank as an expert and activates ranks one by one as needed. Knowledge shared across tasks is no longer trapped inside lone adapters.
Catalog record
| CREATOR | Ziyu Zhao, Yixiao Zhou, Zhi Zhang, Didi Zhu, Tao Shen, Zexi Li, Jinluan Yang, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu Yu Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15103