Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules

Yilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding, Volker Tresp · 2025

Read Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules by Yilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding, Volker Tresp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02587.txt.

Description

Yilun Liu and five co-authors study how to fine tune mixture of experts language models with few added weights, in a 2025 paper. Such models route each input to a subset of expert parts, which makes them efficient. Standard low cost tuning methods ignore that routing, treating the model as if it were uniform.

The authors ask whether the small added tuning modules should also use routing, to match the model's design. They analyze how core parts behave under low cost tuning and test varied routing setups. Trials on two expert models, across common sense and math tasks, favored the routed approach. The paper maps which settings work best in each scenario and gives practical guidance.

Catalog record

CREATORYilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding, Volker Tresp
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02587

Browse the full library · About the Tower of Babel