RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
Mengyang Sun, Maochuan Dou, Tao Feng, Dan Zhang, [0.6em] Yihao Wang, Junpeng Liu, Yifan Zhu, Jie Tang, [0.6em] Tsinghua University · 2026
Read RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism by Mengyang Sun, Maochuan Dou, Tao Feng, Dan Zhang, [0.6em] Yihao Wang, Junpeng Liu, Yifan Zhu, Jie Tang, [0.6em] Tsinghua University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25565.txt.
Description
Mengyang Sun and colleagues strengthen low-rank expert mixtures by rotating experts, not just rescaling them. Standard mixture gates apply a single scalar weight to each chosen expert, capping how far adapters can specialize for varied knowledge. RotMoLE adds a rotation gate per selected expert, exploiting the low-rank structure to let experts specialize more fully, especially when few candidates exist. Tests on complex multi-task and multilingual training confirm the gain over scalar gating.
Catalog record
| CREATOR | Mengyang Sun, Maochuan Dou, Tao Feng, Dan Zhang, [0.6em] Yihao Wang, Junpeng Liu, Yifan Zhu, Jie Tang, [0.6em] Tsinghua University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53350 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25565