Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning

Meng Lou, Hanzhong Guo, Linwei Chen, Yizhou Yu · 2026

Read Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning by Meng Lou, Hanzhong Guo, Linwei Chen, Yizhou Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09640.txt.

Introduction

Lou and three colleagues test whether reward based tuning can beat catastrophic forgetting in visual continual learning. Past work hints reward tuning resists forgetting better than supervised tuning. But whether it wins in hard class incremental and domain incremental setups stayed open. A pilot study confirms reward tuning beats supervised tuning yet still forgets a lot. The authors trace the snag to trajectory level drift blindness.

Among rollouts earning equal rewards, drift from the prior task policy varies widely. That spread tracks forgetting closely across sequential tasks. Their Retention aware Policy Optimization method, RaPO, counters it with trajectory level reward shaping. A retention reward turns drift into a steady signal favoring knowledge keeping rollouts. Cross task advantage normalization steadies training across task borders. Across five visual continual setups, RaPO leads, cutting forgetting while keeping plasticity.

Catalog record

CREATORMeng Lou, Hanzhong Guo, Linwei Chen, Yizhou Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09640

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning Meng Lou, Hanzhong Guo, Linwei Chen, Yizhou Yu Abstract Recent studies suggest that Reinforcement Fine-Tuning (RFT) is inherently more resilient to catastrophic forgetting than Supervised Fine-Tuning (SFT). However, whether RFT (e.g., GRPO) can effectively overcome forgetting in challenging visual continual learning settings, such as class-incremental learning (CIL) and domain-incremental learning (DIL), remains an open problem. Through a pilot study, we confirm that while RFT consistently outperforms SFT, it still suffers from non-negligible forgetting. We empirically trace this bottleneck to Trajectory-level Drift Agnosticism: among candidate rollouts achieving identical task rewards, the KL divergence from the preceding-task policy varies substantially, which strongly correlates with catastrophic forgetting across sequential tasks. Motivated by this insight, we propose Retention-aware Policy Optimization (RaPO), a simple yet effective RFT method that explicitly mitigates forgetting through trajectory-level reward shaping. Specifically, RaPO comprises two core components: (1) Retention Reward that converts trajectory-level distribution drift into a continuous reward signal, preferentially reinforcing knowledge-preserving rollouts within each group; (2) Cross-Task Advantage Normalization (CTAN), which maintains a persistent exponential moving average of reward statistics across task boundaries to stabilize the optimization progress during continual learning. Leveraging the free-form textual generalization of MLLMs, we comprehensively evaluate RaPO across five visual continual learning settings. Extensive experiments demonstrate that RaPO achieves leading performance, substantially reducing catastrophic forgetting while preserving strong plasticity. To the best of our knowledge, this work represents the first systematic exploration of RFT in visual continual learning, offering insights that we hope will inspire future research. Code will be publicly available at: https://github.com/LMMMEng/RaPO . 1 Introduction Reinforcement Fine-Tuning (RFT) with verifiable rewards team2025kimi15 ; bai2025qwen3vl ; guo2025seed15vl ; team2026qwen35 ; zhang2025rlsurvey ; guo2026leveraging has demonstrated remarkable progress in eliciting reasoning capabilities in Multi-modal Large Language Models (MLLMs), outperforming classical Supervised Fine-Tuning (SFT). GRPO shao2024deepseekmath stands as a representative work that leverages verifiable RFT to train powerful large reasoning models guo2025deepseekr1 ; liu2025deepseekv32 . Motivated by this success, several subsequent works liu2025visualrft ; li20252025thinkornot ; tan2025reasonrft ; he2026finer1 have demonstrated that RFT can effectively improve vision tasks even under few-shot training regimes. However, real-world applications frequently encounter streaming data within continuously evolving environments gomes2017survey , requiring large models to continually adapt to newly arriving data without suffering from catastrophic forgetting zheng2025towards . Although numerous SFT-centered approaches shi2025continual ; yang2025recent ; he2026continualinstruction have been developed for continual learning, recent studies lai2025rlnaturally ; shenfeld2026rl_razor have revealed that RFT is naturally more resilient to catastrophic forgetting than SFT. This property stems from the fact that on-policy learning in RFT implicitly biases the optimization toward solutions residing in low-drift distribution spaces, whereas SFT is prone to converge on solutions within arbitrary distribution drift lai2025rlnaturally . Nevertheless, the efficacy of RFT in challenging visual continual learning, such as class-incremental learning (CIL) zhou2024cilreview and domain-incremental learning (DIL) wang2024clcomprehensive , remains an open problem. Figure 1: (a) RFT clearly outperforms SFT in rehearsal-free CIL, but still suffers from significant forgetting. (b) Among equally-rewarded rollouts, KL divergence from the policy in the preceding task varies substantially, and this difference enlarges as tasks progress. To investigate this, we conduct a pilot study on a challenging rehearsal-free few-shot CIL setting on the widely adopted ImageNet-R dataset imagenet-R . Specifically, 200 image classes are randomly split over 10 non-overlapping tasks, with 20 classes and only 5 labeled examples per class at each task. We compare GRPO and SFT based on the Qwen2-VL-2B model wang2024qwen2 , together with a joint-training upper bound. As shown in Figure 1 (a), GRPO consistently outperforms SFT, confirming that the forgetting resilience of RFT lai2025rlnaturally ; shenfeld2026rl_razor successfully transfers to visual tasks. Nevertheless, GRPO still suffers from non-negligible forgetting, demonstrating that it remains insufficient to mitigate stability-plasticity tension in challenging visual continual learning . To further explore the rationale of this phenomenon, we analyze the trajectory-level learning patterns of GRPO during CIL. Specifically, we measure the distribution drift of each rollout generated by the current policy \pi_{t} as its token-level KL divergence from the frozen preceding-task policy \pi_{t-1} . We focus only on rollout groups with maximal task reward so that the comparison isolates drift differences among equal-reward trajectories. As illustrated in Figure 1 (b), candidate rollouts achieving the same task reward exhibit vastly different KL divergence values , a phenomenon we term trajectory-level drift agnosticism . For instance, two distinct trajectories generated from the same input can solve the current task equally well, but exhibit entirely different magnitudes of distributional drift. This discrepancy is increasingly pronounced as the task sequence progresses, which clearly correlates with the accuracy degradation trend shown in Figure 1 (a). This suggests that purely task-reward-driven behavior leads to drift-agnostic credit assignment, which may contribute to severe forgetting. To validate this hypothesis, we design two simple GRPO variants that intervene on all-correct rollout groups in the later tasks, when forgetting becomes pronounced. Specifically, Variant#1 assigns zero reward to any rollout whose KL exceeds the group mean, thereby retaining positive reinforcement only for low-drift trajectories. Conversely, Variant#2 mirrors this operation, preserving positive rewards solely for heavily drifted rollouts. As demonstrated in Figure 1 (a), these variants exhibit opposed behaviors: Variant#1 effectively mitigates forgetting, whereas Variant#2 significantly exacerbates it. Collectively, these results validate trajectory-level drift agnosticism as a key empirical phenomenon in vanilla GRPO, i.e., trajectory-level drift with respect to the preceding-task policy is an actionable signal closely tied to forgetting . However, these hard-thresholding variants are impractical. First, binary gating collapses fine-grained credit assignment required for complex tasks such as dense predictions. Second, it is unstable under small rollout numbers, where even marginal KL differences may trigger winner-take-all updates. Driven by the above observations, we propose Retention-aware Policy Optimization (RaPO), a simple yet effective RFT method that mitigates catastrophic forgetting through trajectory-level reward shaping. RaPO consists of two complementary components. First, a Retention Reward converts the trajectory-level drift from the preceding-task policy into a dense reward signal: rollouts that stay closer to the preceding policy receive proportionally higher rewards. This design differs fundamentally from standard GRPO, i.e., when two rollouts achieve comparable task rewards but exhibit different degrees of drift, RaPO explicitly reinforces the knowledge-preserving one, steering the policy toward regions that adapt to new data while remaining anchored to previously acquired knowledge. Second, Cross-Task Advantage Normalization (CTAN) maintains a persistent smoother of the reward scale, preventing the abrupt advantage fluctuations that arise from sharp reward-distribution shifts at task boundaries. Together, the Retention Reward directs credit assignment toward low-drift trajectories, while CTAN smoothly stabilizes its scale across the continual learning stream. Leveraging the free-form textual generalization capabilities of MLLMs, our method has been comprehensively evaluated across a diverse suite of visual continual learning tasks, including class-incremental image classification, domain-incremental image classification, class-incremental object detection, domain-incremental object detection, and class-incremental video classification. Extensive experimental results in Section 4 demonstrate the promising performance and generalization capacity of RaPO. Overall, our goal is not to chase state-of-the-art performance on different benchmarks, but to systematically explore the potential of verifier-based RFT for visual continual learning. We hope this work will stimulate further research on RFT-based continual learning. 2 Related Work Reinforcement Fine-Tuning (RFT) has demonstrated a superior capacity to incentivize reasoning capabilities in LLMs jaech2024openaio1 . A foundational paradigm is reinforcement

Read the complete text