Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning

Haodong Lu Chongyang Zhao Minhui Xue, Lina Yao Kristen Moore Dong Gong · 2024

Read Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning by Haodong Lu Chongyang Zhao Minhui Xue, Lina Yao Kristen Moore Dong Gong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01004.txt.

Description

Haodong Lu and co-authors present a 2024 study on teaching old AI models new tasks without erasing old skill. The core tension is between bending to new data and holding old ground. Small add-on updates, called LoRA, steer a big model at low cost. The team probe how the size, or rank, of each update shifts that balance, and where in the model the updates sit.

A middle rank won in most spots, but the best rank moved by task and place. Math shows forgetting grows as rank grows. Their method trains each update while pressing its rank down through sparse weights. Across image and language tests, it learned new tasks well and forgot less than past methods. Rank control could become a standard dial in lifelong learning.

Catalog record

CREATORHaodong Lu Chongyang Zhao Minhui Xue, Lina Yao Kristen Moore Dong Gong
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01004

Browse the full library · About the Tower of Babel