Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning
Haodong Lu Chongyang Zhao Minhui Xue, Lina Yao Kristen Moore Dong Gong · 2024
Read Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning by Haodong Lu Chongyang Zhao Minhui Xue, Lina Yao Kristen Moore Dong Gong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01004.txt.
Description
Haodong Lu and co-authors present a 2024 study on teaching old AI models new tasks without erasing old skill. The core tension is between bending to new data and holding old ground. Small add-on updates, called LoRA, steer a big model at low cost. The team probe how the size, or rank, of each update shifts that balance, and where in the model the updates sit.
A middle rank won in most spots, but the best rank moved by task and place. Math shows forgetting grows as rank grows. Their method trains each update while pressing its rank down through sparse weights. Across image and language tests, it learned new tasks well and forgot less than past methods. Rank control could become a standard dial in lifelong learning.
Catalog record
| CREATOR | Haodong Lu Chongyang Zhao Minhui Xue, Lina Yao Kristen Moore Dong Gong |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01004