RL’s Razor: Why Online Reinforcement Learning Forgets Less
Idan Shenfeld, Jyothish PariPulkit Agrawal Improbable AI Lab, MIT {idanshen, jyop, pulkitag}@mit.edu · 2025
Read RL’s Razor: Why Online Reinforcement Learning Forgets Less by Idan Shenfeld, Jyothish PariPulkit Agrawal Improbable AI Lab, MIT {idanshen, jyop, pulkitag}@mit.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04259.txt.
Introduction
Idan Shenfeld and two co-authors explain why reinforcement learning forgets old skills less than ordinary fine-tuning in a 2025 research paper. Both methods can teach a model a new task equally well, yet supervised fine-tuning erodes earlier abilities far more. The gap tracks how far the tuned model's behavior drifts from the original, measured by KL-divergence, a standard distance between probability patterns.
Learning from its own live attempts biases reinforcement learning toward the closest working solution, the authors show. Supervised training faces no such pull and can land far away. Experiments span large language models and robot foundation models, with theory backing the pattern. They name the principle RL's Razor, after the preference for the nearest adequate answer.
Catalog record
| CREATOR | Idan Shenfeld, Jyothish PariPulkit Agrawal Improbable AI Lab, MIT {idanshen, jyop, pulkitag}@mit.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-04259
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RL’s Razor: Why Online Reinforcement Learning Forgets Less Idan Shenfeld, Jyothish PariPulkit Agrawal Improbable AI Lab, MIT {idanshen, jyop, pulkitag}@mit.edu Abstract Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to distributions arbitrarily far from the base model. We validate these findings through experiments with large language models and robotic foundation models and further provide theoretical justification for why on-policy RL updates lead to a smaller KL change. We term this principle RL’s Razor : among all ways to solve a new task, RL prefers those closest in KL to the original model. Our website is available at http://jyopari.github.io/posts/rl_razor . 1 1 footnotetext: Equal contribution. Correspondence to: Jyothish Pari ( [email protected] ), Idan Shenfeld ( [email protected] ). Figure 1: Bias toward KL-minimal solutions reduces forgetting. Left: Among policies that solve the new task, RL converges to those closest in KL to the base model. Right: This KL bias yields higher prior-task retention at matched new-task performance compared to SFT. 1 Introduction Foundation models have rapidly become the backbone of modern AI, powering applications in language, vision, robotics, and beyond. Despite their remarkable capabilities, today’s models are largely static once deployed: they excel at tasks learned during pre-training or post-training, but are not designed to self-improve and continually acquire new capabilities. We imagine a future where deployed models are long-lived agents assisting humans in the long-term and continuously adapting to new needs. As such, models must improve and adapt to new data, environments, and objectives Gao et al. 2025 ; Dao & Le 2025 ; Moradi et al. 2025 ; Li et al. 2025b ; Simonds & Yoshiyama 2025 ; Zweiger et al. 2025 . A central challenge to this vision is catastrophic forgetting —the tendency for models to lose previously acquired capabilities when trained on new tasks McCloskey & Cohen 1989 ; French 1999 ; Kirkpatrick et al. 2017 ; Luo et al. 2023 . Although scaling model size and pre-training data improves robustness Ramasesh et al. 2021 ; Luo et al. 2023 ; Cossu et al. 2024 , catastrophic forgetting remains a persistent obstacle, undermining the promise of continual improvement Bommasani 2021 ; Guo et al. 2025b ; Zweiger et al. 2025 . To enable foundation models to serve as long-term agents, we need to develop post-training methods that allow models to acquire new skills without erasing old ones. To further this goal, we analyze the performance of two widely used post-training schemes of supervised fine-tuning (SFT) and reinforcement learning (RL). Our experiments reveal a surprising finding: even when SFT and RL achieve the same performance on the new task, we observe that SFT often achieves new-task gains by erasing prior knowledge, while RL better preserves old skills . Figure 1 (right) illustrates this tradeoff: although both methods can reach high performance on the new task, RL maintains substantially higher performance on prior tasks compared to SFT. This striking empirical gap raises the question: what underlying mechanism allows RL to improve on new tasks, but unlike SFT, minimally impacts the model’s prior knowledge? Previous approaches to catastrophic forgetting targeted specific factors such as constraining weight updates ( Kirkpatrick et al. 2017 ; Aljundi et al. 2018 ; Zenke et al. 2017 ) , preserving learned features ( Rannen et al. 2017 ; Hou et al. 2019 ) , or regularizing shift in output distribution ( Li & Hoiem 2017 ; Stiennon et al. 2020 ) . While these methods can reduce forgetting, they focus on its effects rather than its underlying cause. Consequently, it remains unclear what truly governs forgetting or why different training algorithms behave so differently. Some prior work claimed that forgetting can be determined by how much the model’s distribution shifts on past tasks ( Rebuffi et al. 2017 ; Castro et al. 2018 ; Chaudhry et al. 2018 ; Wu et al. 2019 ) . Yet in practice, this is infeasible to measure in foundation models, where the set of prior tasks is vast or even unbounded. To search for a more useful principle, we systematically ablated many candidate variables. Surprisingly, we find that forgetting can instead be predicted using only the new task distribution. Specifically, we uncover an empirical forgetting law : When fine-tuning a model \bm{\pi} on a new task \bm{\tau} , the degree of forgetting is accurately predicted by \bm{\mathbb{E}_{x\sim\tau}\big[\text{KL}(\pi_{0}||\pi)\big]} , the KL divergence between the fine-tuned and base policy evaluated on the new task. This law is practically useful since it can be measured, and even influenced, during fine-tuning, without requiring access to past-task data. Although the mechanism remains to be fully understood, the consistency of this law across models and domains suggests it reflects a fundamental property of forgetting. This law also clarifies the surprising difference between SFT and RL. Our analysis reveals a simple but powerful principle we call RL’s Razor : among the many high-reward solutions for a new task, on-policy methods such as RL are inherently biased toward solutions that remain closer to the original policy in KL divergence . Figure 1 (left) highlights this effect: among the many policies that reach a high success rate on the new task, RL is biased toward KL-minimal solutions, while SFT can converge to distant ones. This bias arises directly from RL’s on-policy training : by sampling from the model’s own distribution at every step, RL constrains learning to outputs already given non-negligible probability by the base model. To improve reward, these samples are reweighted and used to update the model, which gradually shifts the policy rather than pulling it toward an arbitrary distribution. Thus, when multiple equally good solutions exist for a new task, RL tends to find solutions close to the original policy, while SFT can converge to solutions much farther away, depending on the provided labels. Theoretical analysis in a simplified setting confirms this view, showing that policy gradient methods converge to KL-minimal solutions even without explicit regularization. Finally, to validate the KL hypothesis, we construct an “oracle SFT” distribution that provably minimizes KL divergence while achieving perfect accuracy. Training on this oracle distribution produces even less forgetting than RL itself. This demonstrates that RL’s advantage does not stem from being inherently different, but from its implicit KL minimization. Whenever training is biased toward KL-minimal solutions, forgetting is reduced. Our main contributions are: • We show that RL fine-tuning forgets less than SFT, even when both reach the same performance on new tasks. • We uncover an empirical forgetting law: the KL divergence to the base policy, measured on the new task, as a strong predictor of catastrophic forgetting across objectives and hyperparameters. • We provide empirical and theoretical evidence that the on-policy nature of policy gradient methods leads to smaller KL shifts and explains RL’s advantage. Together, these findings suggest a new perspective on