Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates
Parjanya Prajakta Prashant, Jiongli ZhuAldan CreoBabak SalimiUniversity of California San Diego · 2026
Read Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates by Parjanya Prajakta Prashant, Jiongli ZhuAldan CreoBabak SalimiUniversity of California San Diego (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20005.txt.
Description
Parjanya Prajakta Prashant and three co-workers tackle catastrophic forgetting when fine-tuning large language models. They note forgetting per step is bounded by the learning rate times the square root of training loss, so high-loss batches most cause forgetting. Their FINCH schedule therefore lowers the rate of learning on high-loss batches and raises it as the model converges, leaving the objective untouched. Across knowledge, science and low-resource language benchmarks, FINCH cuts forgetting by ninety-three percent on average while matching standard fine-tuning on tasks.
Catalog record
| CREATOR | Parjanya Prajakta Prashant, Jiongli ZhuAldan CreoBabak SalimiUniversity of California San Diego |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68149 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20005