RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism

Mengyang Sun, Maochuan Dou, Tao Feng, Dan Zhang, [0.6em] Yihao Wang, Junpeng Liu, Yifan Zhu, Jie Tang, [0.6em] Tsinghua University · 2026

Read RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism by Mengyang Sun, Maochuan Dou, Tao Feng, Dan Zhang, [0.6em] Yihao Wang, Junpeng Liu, Yifan Zhu, Jie Tang, [0.6em] Tsinghua University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25565.txt.

Description

Mengyang Sun and colleagues strengthen low-rank expert mixtures by rotating experts, not just rescaling them. Standard mixture gates apply a single scalar weight to each chosen expert, capping how far adapters can specialize for varied knowledge. RotMoLE adds a rotation gate per selected expert, exploiting the low-rank structure to let experts specialize more fully, especially when few candidates exist. Tests on complex multi-task and multilingual training confirm the gain over scalar gating.

Catalog record

CREATORMengyang Sun, Maochuan Dou, Tao Feng, Dan Zhang, [0.6em] Yihao Wang, Junpeng Liu, Yifan Zhu, Jie Tang, [0.6em] Tsinghua University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53350
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25565

Browse the full library · About the Tower of Babel