Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback
Yutao Yang, Jie Zhou, Junsong Li, Qianjun Pan, Bihao Zhan, Qin Chen, Xipeng Qiu, Liang He · 2025
Read Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback by Yutao Yang, Jie Zhou, Junsong Li, Qianjun Pan, Bihao Zhan, Qin Chen, Xipeng Qiu, Liang He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09925.txt.
Introduction
Yutao Yang and seven colleagues let a language model keep learning from live human corrections, mistakes and all. Real feedback streams in messy, with wrong labels mixed in. Their framework, RiCL, learns new skills from that stream without forgetting old ones.
Three parts do the work. A purifier spots likely mislabeled examples by checking consistency over time. A preference step aligns the model with what humans meant, blending their notes with model output. A contrastive module builds sturdy features that do not lean on shaky labels. On two relation-extraction benchmarks with realistic noise, RiCL beat rival combinations clearly.
Catalog record
| CREATOR | Yutao Yang, Jie Zhou, Junsong Li, Qianjun Pan, Bihao Zhan, Qin Chen, Xipeng Qiu, Liang He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-09925
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback Yutao Yang, Jie Zhou, Junsong Li, Qianjun Pan, Bihao Zhan, Qin Chen, Xipeng Qiu, Liang He Abstract This paper introduces an interactive continual learning paradigm where AI models dynamically learn new skills from real-time human feedback while retaining prior knowledge. This paradigm distinctively addresses two major limitations of traditional continual learning: (1) dynamic model updates using streaming, real-time human-annotated data, rather than static datasets with fixed labels, and (2) the assumption of clean labels, by explicitly handling the noisy feedback common in real-world interactions. To tackle these problems, we propose RiCL , a Reinforced interactive Continual Learning framework leveraging Large Language Models (LLMs) to learn new skills effectively from dynamic feedback. RiCL incorporates three key components: a temporal consistency-aware purifier to automatically discern clean from noisy samples in data streams; an interaction-aware direct preference optimization strategy to align model behavior with human intent by reconciling AI-generated and human-provided feedback; and a noise-resistant contrastive learning module that captures robust representations by exploiting inherent data relationships, thus avoiding reliance on potentially unreliable labels. Extensive experiments on two benchmark datasets (FewRel and Tacred), contaminated with realistic noise patterns, demonstrate that our RiCL approach substantially outperforms existing combinations of state-of-the-art online continual learning and noisy-label learning methods. 1 Introduction Figure 1: The process of Interactive Continual Learning. Continual learning (CL) has emerged as a critical paradigm for enabling artificial intelligence (AI) systems to continuously adapt and evolve in dynamic environments [ 1 ] . Unlike traditional machine learning approaches, which often rely on static datasets and fixed task boundaries, continual learning emphasizes incremental knowledge acquisition while preserving previously learned capabilities [ 2 , 3 , 4 ] . Recent advancements in AI, particularly large language models (LLMs), have further highlighted the potential of interactive and adaptive systems that learn through real-world interactions [ 5 , 6 , 7 ] . For real-world interactive scenarios (e.g., personalized assistance, autonomous systems, or adaptive decision-making platforms), the model outputs the predicted labels to the human, and the human gives feedback to update the model (as shown in Figure 1 ). Most traditional continual learning approaches often rely on pre-collected static datasets with given labels [ 8 ] . They mainly focus on reducing the catastrophic forgetting problem via replay-based [ 9 ] , regularization-based [ 10 ] , and architecture-based [ 11 ] methods. Recently, some studies have focused on online continual learning that learns new knowledge with data streaming to adapt in real-time [ 12 , 13 , 14 ] . Qi et al. [ 15 ] introduced an interactive continual learning framework that enables collaborative interactions among multiple models of varying sizes. However, few studies have explored continuous learning from human-machine interaction , with a focus on dynamic model adaptation based on noisy human feedback. These methods face two significant challenges prevalent in real-world interactive scenarios: (1) Evolving over time via the data stream from real-time human feedback. As illustrated in Figure 1 , this task involves feeding a continuous data stream into LLMs while incorporating real-time human feedback to refine the model’s performance without forgetting the previous abilities; (2) Robustly learning from inherently noisy human feedback, which is a common characteristic of interactive applications. In these scenarios, humans provide corrections to the predictions made by LLMs, which may sometimes be incorrect. Furthermore, existing studies often overlook the importance of leveraging the predicted results for further improvement. These limitations impede the effective deployment of AI systems in environments where human-in-the-loop interactions and dynamic, real-time data updates are critical. To bridge this gap, we propose RiCL , a Reinforced interactive Continual Learning framework grounded in LLMs, designed to harmonize real-time human feedback with stable knowledge retention. First, we design a temporal consistency-aware purifier (TCP), which dynamically discriminates between clean and noisy samples in streaming data by analyzing temporal coherence and prediction stability, enabling selective integration of reliable information. Then, we propose an interaction-aware direct preference optimization (IPO), which aligns model behavior with human intent by resolving conflicts between AI-generated outputs and human feedback, leveraging a reinforcement learning paradigm to prioritize feedback that enhances task coherence and user satisfaction. Finally, we introduce a noise-resistant contrastive learning module that extracts robust feature representations by exploiting inherent semantic relationships in the data, reducing reliance on potentially noisy labels through self-supervised alignment of augmented samples. Our experiments on two datasets (Fewrel and Tacred) contaminated with real-world noise patterns demonstrate that RiCL outperforms state-of-the-art combinations of online CL and noisy-label learning methods. By bridging the gap between theoretical lifelong learning principles and practical interactive AI systems, this work advances the development of autonomous, adaptive models capable of thriving in open-world environments. In summary, our primary contributions are threefold: • To learn like a human, we propose a reinforced interactive continual learning framework to learn new knowledge without forgetting previous skills in real time via noisy human feedback from human-machine interaction. • We design an interaction-aware direct preference optimization and a noise-resistant contrastive learning strategy to learn continually from clean and noisy datasets that are indicated by a temporal consistency-aware purifier. • Our extensive experiments demonstrate that RiCL obtains better performance in both mitigating catastrophic forgetting and handling label noise, offering actionable insights for robust continual learning in real-world streaming interactive scenarios. 2 Related Work Continual Learning Continual learning enables artificial intelligence models to incrementally learn new tasks without forgetting previously acquired knowledge [ 7 ] . The literature on continual learning predominantly categorizes methods into three classes: rehearsal-based [ 9 ] , regularization-based [ 16 ] , and parameter-isolation [ 11 ] approaches. Rehearsal-based methods preserve previously learned knowledge by retaining subsets of past samples or generating pseudo-samples (generative replay) to periodically revisit during new task learning, mitigating catastrophic forgetting [ 17 , 18 , 19 ] . For example, LAMOL [ 20 ] generates pseudo-samples to rehearse past NLP tasks, and RVAE_LAMOL [ 21 ] utilizes a variational autoencoder for stable and accurate pseudo-sample generation. Regularization-based approaches impose constraints or penalties on parameter updates to balance new learning and knowledge retention, preventing significant deviations from previously learned representations [ 22 ] . For instance, Continual Proximal Policy Optimization (CPPO) [ 23 ] integrates sample-wise weighting into policy optimization to maintain prior knowledge while learning new policies effectively. Parameter-isolation methods allocate separate parameters or subnetworks for distinct tasks, explicitly avoiding interference among tasks [ 24 ] . JARe [ 25 ] dynamically utilizes task-related knowledge retrieval to effectively manage parameter adjustments for different downstream tasks. Notably, Qi et al. [ 15 ] proposed Interactive Continual Learning (ICL), which leverages interactions between different model complexities to enhance memory retrieval and collaborative reasoning capabilities. Moreover, Xu et al. [ 26 ] utilized reinforcement learning to dynamically optimize neural architectures for continual learning. Unlike these studies, we focus on an interactive continual learning task to learn real-time human feedback from interactions, prioritizing alignment between human intent and model output. Online Continual Learning Online continual learning explores scenarios where models continuously adapt to new knowledge by sequentially or concurrently processing incoming data streams [ 27 , 28 ] . Studies increasingly focus on effective knowledge retention and adaptation in realistic, dynamically evolving scenarios [ 29 ]