S’MoRE : Structural Mixture of Residual Experts for LLM Fine-tuning
Hanqing Zeng, Yinglong Xia, Zhuokai Zhao, Gilbert Jiang, Qiang Zhang, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Benyu Zhang · 2025
Read S’MoRE : Structural Mixture of Residual Experts for LLM Fine-tuning by Hanqing Zeng, Yinglong Xia, Zhuokai Zhao, Gilbert Jiang, Qiang Zhang, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Benyu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06426.txt.
Description
Hanqing Zeng and co-authors propose, in a 2025 paper, a leaner way to fine-tune large language models. Fine-tuning must balance two pulls: change few settings, to stay cheap, yet keep enough capacity to learn the task. A popular low-rank method is cheap but rigid. Expert-mixture designs are flexible but wasteful. Their method, S'MoRE, breaks expert weights into layered low-rank leftovers, then routes each input token through a tree of those pieces.
A few small matrices, assembled on the fly, can stand in for a crowd of experts. The team casts the flow between layers as a graph network and proves the design's flexibility grows steeply with size for the same budget. Tests back the claim: fine-tuning quality beat both parent methods. Code is public for others to try.
Catalog record
| CREATOR | Hanqing Zeng, Yinglong Xia, Zhuokai Zhao, Gilbert Jiang, Qiang Zhang, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Benyu Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06426