Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs
Alessio Quercia, Arya Bangun, Ira Assent, Hanno Scharr · 2026
Read Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs by Alessio Quercia, Arya Bangun, Ira Assent, Hanno Scharr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03493.txt.
Description
Fine-tuning a large model with LoRA raises a trade-off: adapt well to the new task, or forget less of the old knowledge. Alessio Quercia and 3 co-authors analyze that balance in a 2026 paper through the lens of principal components of the weight matrices. Existing methods initialize adaptation from the first components or the last, with inconsistent advice.
Their analysis finds that the intermediate components hit the sweet spot. Tuning them balances task gains against forgetting and tolerates higher learning rates better than the extremes. The authors turn the finding into practical initialization guidelines for LoRA-style methods. A broad empirical study across vision and language tasks confirms higher accuracy with reduced forgetting.
Catalog record
| CREATOR | Alessio Quercia, Arya Bangun, Ira Assent, Hanno Scharr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-03493