Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates

Yibo Li, Zijie Lin, Ailin Deng, Xuan Zhang, Yufei He, Shuo Ji, Tri Cao, Bryan Hooi · 2026

Read Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates by Yibo Li, Zijie Lin, Ailin Deng, Xuan Zhang, Yufei He, Shuo Ji, Tri Cao, Bryan Hooi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18510.txt.

Introduction

Yibo Li and seven colleagues give language model agents a way to learn on the job with no new training runs. An agent's weights freeze once it ships, so it cannot learn from its own past wins and slips in the plain way. Full re-training is dear and can wipe old skills. The team skip the weight change and learn in a leaner way.

Their plan feeds the agent fresh cues drawn from its own past runs, right at the time of need. Those cues steer its next steps while its weights stay put. In time the agent acts as if it had learned, with no risk of a bad train run. The trick could let field agents grow with use at low cost. It also keeps the base model whole for the next user.

Catalog record

CREATORYibo Li, Zijie Lin, Ailin Deng, Xuan Zhang, Yufei He, Shuo Ji, Tri Cao, Bryan Hooi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18510

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates Yibo Li, Zijie Lin, Ailin Deng, Xuan Zhang, Yufei He, Shuo Ji, Tri Cao, Bryan Hooi Abstract While Large Language Model (LLM) agents excel at general tasks, they inherently struggle with continual adaptation due to the frozen weights after deployment. Conventional reinforcement learning (RL) offers a solution but incurs prohibitive computational costs and the risk of catastrophic forgetting. We introduce Just-In-Time Reinforcement Learning (JitRL) , a training-free framework that enables test-time policy optimization without any gradient updates. JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly. These estimates are then used to directly modulate the LLM’s output logits. We theoretically prove that this additive update rule is the exact closed-form solution to the KL-constrained policy optimization objective. Extensive experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods. Crucially, JitRL outperforms the performance of computationally expensive fine-tuning methods (e.g., WebRL) while reducing monetary costs by over 30 × \times , offering a scalable path for continual learning agents. The code is available at https://github.com/liushiliushi/JitRL . 1 Introduction Figure 1 : While standard RL performs policy gradient updates during training using previous trajectories, JitRL operates at test time. Specifically, it retrieves trajectories relevant to the current state to estimate advantages A , subsequently refining the output logits through a KL-regularized policy optimization objective. While humans can learn “on the fly”, current AI agents fundamentally lack this ability because their weights are frozen after training ( 3 ; 2 ) . This limitation severely restricts their practicality: when deployed in unfamiliar or dynamic settings, AI agents are analogous to newly hired employees without on-the-job training, which often repeatedly make the same errors due to their failure to learn, whereas humans can adapt to their environment, including by learning from their mistakes. Highlighting the importance of this issue, 6 proposes an “AGI Score” assessing progress toward artificial general intelligence and finds that current AI systems are most deficient in their “capability to continually learn new information”. One potential solution is conventional reinforcement learning (RL) ( 13 ; 14 ; 9 ) , performed in a continually policy gradient updating manner. However, this approach requires a large amount of training data to perform well. It is also computationally expensive, due to the high cost of RL and the need for frequent updates to keep the model up to date. In addition, it is prone to catastrophic forgetting, degrading the model’s previously learned capabilities ( 8 ) . Prior work finds that conventional RL yields only limited improvements when evaluated in continual adaptation settings ( 5 ) . An alternative approach to this problem relies on in-context learning (ICL) ( 1 ) to enable the agent to learn at test time ( 18 ; 22 ) . These methods rely on prompt or context engineering to incorporate information from past experiences. However, such approaches tend to struggle as the required context length grows, especially since agentic tasks often involve long sequences of interactions ( 10 ; 25 ) . Crucially, ICL lacks the generality of RL. While prompts are confined to explicit textual descriptions, RL optimizes policies through rewards, enabling the mastery of complex skills that are difficult to articulate in text. In light of these limitations, a fundamental question arises: Can we enable agents to learn continually in a flexible way, without expensive parameter updates? Our approach, Just-In-Time Reinforcement Learning (JitRL), enables general-purpose learning by adopting an RL formulation, while maintaining efficiency by avoiding gradient updates. Instead of gradient updates, JitRL maintains a dynamic memory bank that stores trajectories as <state, action, reward> triplets, as illustrated in Figure 1 . Then, given the agent’s current state, it retrieves relevant trajectories from the memory, and learns from them “just in time.” To do so, it uses these trajectories to estimate the advantage of each action in the current state – i.e., how much better each action is relative to the average. These advantage estimates are then used to adjust the model’s output logits. Crucially, we theoretically prove that our update rule is the exact closed-form solution to the policy optimization objective under a KL constraint. In this way, JitRL enables continual self-evolution without the prohibitive costs of gradient-based training. We empirically validate JitRL on two benchmarks: WebArena ( 26 ) for realistic web navigation and Jericho ( 4 ) for long-horizon text-based games. Extensive experiments demonstrate that JitRL sets a new state-of-the-art, outperforming training-free baselines and weight-update methods, while reducing the monetary costs of conventional RL by over 30×. Furthermore, our results highlight JitRL’s robustness, showing consistent gains across diverse LLM backbones and effective generalization to unseen tasks. 2 Related Work Reinforcement Learning. Reinforcement learning (RL) algorithms like PPO ( 16 ) and GRPO ( 17 ) have effectively aligned LLM agents for complex tasks. This paradigm spans diverse domains, ranging from optimizing search queries in information retrieval ( 7 ; 19 ) to refining tool execution ( 9 ; 14 ) . However, these gradient-based methods suffer from prohibitive computational costs and yield static models, fundamentally limiting their adaptability to distribution shifts. Training-Free Inference Enhancement. Recent works have integrated external memory modules to enhance agents directly at test time. Systems like MemGPT and Generative Agents ( 11 ; 12 ) utilize hierarchical memory to store historical interactions, while frameworks like Voyager and Reflexion ( 21 ; 18 ) retrieve textual descriptions of past skills or failures to improve future performance. Similarly, A-mem ( 24 ) constructs interconnected knowledge networks through dynamic indexing. Rather than merely retrieving text for in-context learning, our method treats memory as a non-parametric policy distribution. We perform soft updates directly on the LLM’s logits, effectively achieving policy improvement without the overhead of parameter updates. 3 Preliminaries Reinforcement Learning (RL) aims to optimize a policy \pi_{\theta} , which maps a given state to a probability distribution over actions, to maximize the expected cumulative reward. The objective function is defined as J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}[R(\tau)] , where \tau=(s_{0},a_{0},r_{0},s_{1},a_{1},r_{1}\dots,s_{n},a_{n},r_{n}) represents a trajectory of n interactions, and R(\tau) denotes the cumulative discounted return of that trajectory. A fundamental approach to optimize this objective is the Policy Gradient method ( 23 ; 20 ) . Intuitively, this method shifts the probability mass towards actions that yield higher returns by updating parameters in the direction of the gradient: \nabla_{\theta}J(\theta)=\mathbb{E}_{s,a\sim\pi_{\theta}}[\nabla_{\theta}\log\pi_{\theta}(a|s)\cdot A(s,a)], (1) where A(s,a) is the advantage function . The advantage function serves as a critic to evaluate the selected action relative to a baseline, formulated as: A(s,a)=Q(s,a)-V(s). (2) Here, Q(s,a) is the action-value function , representing the expected return of taking action a in state s , while V(s) is the state-value function , representing the average expected return of being in state s . Thus, A(s,a) quantifies how much better a specific action is compared to the average performance of the policy in that state. Reinforcement Learning typically requires the training of additional value networks to estimate the advantage. This process is computationally expensive and results in a static model that lacks the flexibility to adapt at test time. 4 Method Figure 2 : Overview of the Just-In-Time Reinforcement Learning (JitRL) framework. The system

Read the complete text