Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
Yilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding, Volker Tresp · 2025
Read Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules by Yilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding, Volker Tresp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02587.txt.
Description
Yilun Liu and five co-authors study how to fine tune mixture of experts language models with few added weights, in a 2025 paper. Such models route each input to a subset of expert parts, which makes them efficient. Standard low cost tuning methods ignore that routing, treating the model as if it were uniform.
The authors ask whether the small added tuning modules should also use routing, to match the model's design. They analyze how core parts behave under low cost tuning and test varied routing setups. Trials on two expert models, across common sense and math tasks, favored the routed approach. The paper maps which settings work best in each scenario and gives practical guidance.
Catalog record
| CREATOR | Yilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding, Volker Tresp |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02587