Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs

Alessio Quercia, Arya Bangun, Ira Assent, Hanno Scharr · 2026

Read Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs by Alessio Quercia, Arya Bangun, Ira Assent, Hanno Scharr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03493.txt.

Description

Fine-tuning a large model with LoRA raises a trade-off: adapt well to the new task, or forget less of the old knowledge. Alessio Quercia and 3 co-authors analyze that balance in a 2026 paper through the lens of principal components of the weight matrices. Existing methods initialize adaptation from the first components or the last, with inconsistent advice.

Their analysis finds that the intermediate components hit the sweet spot. Tuning them balances task gains against forgetting and tolerates higher learning rates better than the extremes. The authors turn the finding into practical initialization guidelines for LoRA-style methods. A broad empirical study across vision and language tasks confirms higher accuracy with reduced forgetting.

Catalog record

CREATORAlessio Quercia, Arya Bangun, Ira Assent, Hanno Scharr
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03493

Browse the full library · About the Tower of Babel