Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors

Guozheng Ma, Lu Li, Zilin Wang, Pierre-Luc Bacon, Dacheng Tao · 2026

Read Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors by Guozheng Ma, Lu Li, Zilin Wang, Pierre-Luc Bacon, Dacheng Tao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25527.txt.

Introduction

Modern reinforcement learning agents almost always start from something learned offline: a dataset, a pretrained policy, or a foundation model. Guozheng Ma and colleagues argue in 2026 that no universal rule governs how an agent should treat that prior knowledge. Prior quality varies by deployment and shifts during training itself, so the same choice that helps in one setting hurts in another. Benchmark rankings, they contend, cannot settle questions that are local by nature.

Their alternative is diagnosis driven management: watch deployment specific evidence and adjust the learner's reliance on its priors as training proceeds. A framework sorts prior effects into three functional roles inside online optimisation. Controlled experiments show clean help or hurt reversals when the setting changes, and evidence ranges from model post training to embodied robots. Five serious counterarguments are addressed head on.

Catalog record

CREATORGuozheng Ma, Lu Li, Zilin Wang, Pierre-Luc Bacon, Dacheng Tao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25527

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors Guozheng Ma, Lu Li, Zilin Wang, Pierre-Luc Bacon, Dacheng Tao Abstract Online reinforcement learning (RL) agents increasingly depend on knowledge acquired offline to achieve practical efficiency. Originally studied in offline-to-online RL, this paradigm now spans foundation model post-training and embodied intelligence, with prior types expanding from offline datasets and pre-trained policies to increasingly diverse knowledge sources such as multimodal foundation models and generative world models. Offline priors have become central to how deep RL is developed and deployed. However, this reliance introduces a challenge that the prevailing benchmark-driven paradigm cannot resolve: because prior validity varies across deployments and shifts during training, no single approach to managing it is universally optimal, and benchmark rankings offer limited guidance for real-world deployments. Rather than pursuing universal solutions, we argue that the field should shift to diagnosis-driven tension management, in which deployment-specific evidence guides how the learner relates to its priors throughout training, enabling both flexible and adaptive deployment. We support this position with a framework characterizing how priors reshape online optimization through three functional roles, controlled experiments demonstrating help-or-hurt reversals, cross-domain evidence from foundation model post-training to embodied intelligence, and engagement with five substantive counterarguments. \titlecontents section[0em] \thecontentslabel \contentspage 1 Introduction Online reinforcement learning (RL) agents increasingly rely on knowledge acquired offline to achieve practical performance. In foundation model post-training, large-scale pre-trained models provide the base capability that online RL refines for alignment, reasoning, and agentic applications [ 55 , 15 ] . In embodied intelligence, simulators, demonstration datasets, world models, and pre-trained policies supply the prior knowledge that makes online learning on physical platforms feasible [ 58 , 18 , 67 , 40 ] . Despite differences in domain, objective, and prior type, offline priors now provide the foundation from which agents learn through online experience [ 6 , 77 , 60 , 34 ] . Despite these benefits, how to use offline priors remains a persistent challenge: the same reliance decision that helps in one setting often hurts in another. In offline-to-online RL, whether to preserve the pre-trained policy, retain the offline dataset, or maintain conservative value estimates each produces different outcomes depending on the quality of the offline sources and their relationship to the deployment task [ 35 , 77 , 6 , 68 ] . In foundation model post-training, the role of the reference constraint varies with the fidelity of the reward signal: when rewards are verifiable, strong regularization restricts the discovery of novel strategies [ 73 ] , while when rewards come from learned preference models, the same regularization is essential for preventing overoptimization of proxy scores [ 55 , 13 ] . In robotics, whether to train in simulation and transfer or to learn directly on physical hardware produces different outcomes depending on the fidelity of the simulator and the complexity of the contact dynamics [ 4 , 40 , 33 ] . These inconsistencies recur across communities, domains, and prior types, suggesting a shared structural property rather than isolated engineering issues. We argue that these inconsistencies reflect an inherent conflict: offline priors carry knowledge that is inevitably bounded, while online RL exists precisely to push beyond those bounds. \bullet Specifically, whether the prior takes the form of a pre-trained policy, a simulator, or an offline dataset, it encodes knowledge acquired under conditions that may differ from those of deployment. As a result, the extent to which this knowledge remains useful is uncertain and can only be revealed through online interaction. We call this the bounded commitment of offline priors: valuable knowledge with an inherently uncertain scope of validity. \bullet On the other hand, online RL drives the agent toward optimal performance through interaction with the deployment environment. This requires the agent to eventually surpass what the prior covers, while relying on its knowledge to get there efficiently. As long as the agent operates within the scope of the prior’s validity, reliance is purely beneficial. Once learning pushes beyond that scope, a genuine tension emerges: stronger reliance constrains adaptation, while freer adaptation risks discarding knowledge that still helps elsewhere. Crucially, the agent cannot know where this boundary lies, and the boundary itself shifts as online experience accumulates. Unlike a static trade-off that can be settled at design time, this tension persists and evolves throughout learning. Hence, tension management is a central challenge of this paradigm. One consequence is that tension management has no universal optimum: the right reliance on each prior depends on the deployment and shifts as learning progresses. Despite this, the field continues to evaluate progress by comparing methods on fixed benchmarks, implicitly assuming that the resulting rankings reflect universal truths rather than deployment-specific matches. The field accumulates condition-specific performance rankings rather than transferable understanding. The way forward requires a shift in research perspective. Rather than asking which method is best, the field should ask what determines when each design choice helps or hurts. Figure 1: Online RL from scratch (top) versus with offline priors (bottom). The agent-environment interaction loop is identical in both cases. Offline priors add knowledge sources that accelerate learning but also introduce structural tensions analyzed in 3 10 . \state We argue that the field of online RL with offline priors should move beyond one-size-fits-all methods toward diagnosis-driven tension management. Since no universal optimum exists for how agents should rely on their priors, effective deployment requires diagnostic infrastructure that can assess the prior-deployment match and monitor how tensions evolve during learning, enabling both flexible and adaptive deployment. We develop this position as follows. 4 9 defines the paradigm and introduces a taxonomy of offline prior types. 5 10 analyzes why offline priors sharpen the core tensions of online RL and introduces the concept of bounded commitments. 6 13 presents evidence that tension management has no universal optimum and argues for a shift from benchmark-driven to diagnosis-driven tension management. 7 17 engages with several potential objections and counterarguments to our core position. 8 18 concludes with research opportunities and a broader perspective. 9 The Paradigm of Online RL with Offline Priors Instead of learning from scratch , online RL agents increasingly learn not only from their own online interaction with the environment but also from knowledge acquired offline: setting initial parameters, constraining how far the learner may deviate from prior behavior, or supplying supplementary data or predictions alongside real interaction. Though studied under different names across offline-to-online RL [ 48 , 49 ] , LLM post-training [ 55 , 15 ] , sim-to-real transfer [ 66 , 4 ] , model-based RL [ 18 , 20 ] , and vision-language-action model fine-tuning [ 78 , 16 ] , these mechanisms define a common paradigm that we call online RL with offline priors . As in any RL system, the interaction loop has two sides: an agent that selects actions and an environment that produces states and rewards. Offline knowledge can concern either side, and we organize the resulting priors accordingly. Table 1: Taxonomy of offline priors in online RL, classified by what knowledge they encode and how they function during online learning. Agent-side priors encode knowledge about how to act or evaluate; environment-side priors encode knowledge about how the world behaves. Knowledge Side

Read the complete text