SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning

Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ho Shing Ip, Sam Kwong · 2025

Read SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning by Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ho Shing Ip, Sam Kwong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02486.txt.

Description

Jinpeng Chen and six co-authors split forgetting in ever learning vision language models into two kinds in a 2025 paper. As such models take new tasks in turns, old skills fade. Some fading is skin deep: the facts stay, but the answer style drifts to the newest task's dress, which makes old answers unusable. The rest is true loss, where well dressed answers turn wrong.

Their fix works in order. First, training sets are restyled to one shared varied dress, which stops the style drift. Then a regularized low rank method, RegLoRA, locks the weights where old knowledge chiefly sits. The pair, named SEFE, topped past methods on the standard runs. The authors say style must be cleared before true loss can even be seen.

Catalog record

CREATORJinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ho Shing Ip, Sam Kwong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02486

Browse the full library · About the Tower of Babel