HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
Jia Wei, Zhonghao Zhang, Ping Chen, Qianyang Li, Yancheng, Pan, Shaoxun Wang, Ziyi Qiu, Longxiang Wang · 2026
Read HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models by Jia Wei, Zhonghao Zhang, Ping Chen, Qianyang Li, Yancheng, Pan, Shaoxun Wang, Ziyi Qiu, Longxiang Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18795.txt.
Description
Jia Wei and seven colleagues adapt low-rank fine-tuning to mixture-of-experts models, where only some experts fire per token. Their HELLoRA method attaches adapters only to the most used experts in each layer, cutting trainable weights and adapter compute while lifting scores. A stricter variant, HELLoRI, freezes one projection and sparsifies the other. Across three expert models and maths, coding, and safety tasks, HELLoRA beats strong efficient-tuning rivals.
Catalog record
| CREATOR | Jia Wei, Zhonghao Zhang, Ping Chen, Qianyang Li, Yancheng, Pan, Shaoxun Wang, Ziyi Qiu, Longxiang Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45043 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18795