S’MoRE : Structural Mixture of Residual Experts for LLM Fine-tuning

Hanqing Zeng, Yinglong Xia, Zhuokai Zhao, Gilbert Jiang, Qiang Zhang, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Benyu Zhang · 2025

Read S’MoRE : Structural Mixture of Residual Experts for LLM Fine-tuning by Hanqing Zeng, Yinglong Xia, Zhuokai Zhao, Gilbert Jiang, Qiang Zhang, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Benyu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06426.txt.

Description

Hanqing Zeng and co-authors propose, in a 2025 paper, a leaner way to fine-tune large language models. Fine-tuning must balance two pulls: change few settings, to stay cheap, yet keep enough capacity to learn the task. A popular low-rank method is cheap but rigid. Expert-mixture designs are flexible but wasteful. Their method, S'MoRE, breaks expert weights into layered low-rank leftovers, then routes each input token through a tree of those pieces.

A few small matrices, assembled on the fly, can stand in for a crowd of experts. The team casts the flow between layers as a graph network and proves the design's flexibility grows steeply with size for the same budget. Tests back the claim: fine-tuning quality beat both parent methods. Code is public for others to try.

Catalog record

CREATORHanqing Zeng, Yinglong Xia, Zhuokai Zhao, Gilbert Jiang, Qiang Zhang, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Benyu Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06426

Browse the full library · About the Tower of Babel