Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates

Parjanya Prajakta Prashant, Jiongli ZhuAldan CreoBabak SalimiUniversity of California San Diego · 2026

Read Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates by Parjanya Prajakta Prashant, Jiongli ZhuAldan CreoBabak SalimiUniversity of California San Diego (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20005.txt.

Description

Parjanya Prajakta Prashant and three co-workers tackle catastrophic forgetting when fine-tuning large language models. They note forgetting per step is bounded by the learning rate times the square root of training loss, so high-loss batches most cause forgetting. Their FINCH schedule therefore lowers the rate of learning on high-loss batches and raises it as the model converges, leaving the objective untouched. Across knowledge, science and low-resource language benchmarks, FINCH cuts forgetting by ninety-three percent on average while matching standard fine-tuning on tasks.

Catalog record

CREATORParjanya Prajakta Prashant, Jiongli ZhuAldan CreoBabak SalimiUniversity of California San Diego
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68149
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20005

Browse the full library · About the Tower of Babel