HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models

Jia Wei, Zhonghao Zhang, Ping Chen, Qianyang Li, Yancheng, Pan, Shaoxun Wang, Ziyi Qiu, Longxiang Wang · 2026

Read HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models by Jia Wei, Zhonghao Zhang, Ping Chen, Qianyang Li, Yancheng, Pan, Shaoxun Wang, Ziyi Qiu, Longxiang Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18795.txt.

Description

Jia Wei and seven colleagues adapt low-rank fine-tuning to mixture-of-experts models, where only some experts fire per token. Their HELLoRA method attaches adapters only to the most used experts in each layer, cutting trainable weights and adapter compute while lifting scores. A stricter variant, HELLoRI, freezes one projection and sparsifies the other. Across three expert models and maths, coding, and safety tasks, HELLoRA beats strong efficient-tuning rivals.

Catalog record

CREATORJia Wei, Zhonghao Zhang, Ping Chen, Qianyang Li, Yancheng, Pan, Shaoxun Wang, Ziyi Qiu, Longxiang Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45043
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18795

Browse the full library · About the Tower of Babel