Little By Little: Continual Learning via Self-Activated Sparse Mixture-of-Rank Adaptive Learning
Haodong Lu Chongyang Zhao Jason Xue, Lina Yao Kristen Moore Dong Gong · 2025
Read Little By Little: Continual Learning via Self-Activated Sparse Mixture-of-Rank Adaptive Learning by Haodong Lu Chongyang Zhao Jason Xue, Lina Yao Kristen Moore Dong Gong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21035.txt.
Description
Haodong Lu and co-authors propose a way in 2025 for large models to keep learning new tasks without losing old skills. Past methods add big blocks of new parts and freeze the old ones, but the blocks can clash, repeat each other, and confuse the choice of which block to use.
Their method adds tiny rank one parts that act like small memory units, each holding a key and a value. No extra switch is needed, since each unit judges its own fit to the task. Tests on image and text models show less forgetting and better scores than past methods.
Catalog record
| CREATOR | Haodong Lu Chongyang Zhao Jason Xue, Lina Yao Kristen Moore Dong Gong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-21035