TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning
Matthew M. Hong, Jesse Zhang, Anusha Nagabandi, Abhishek Gupta, [4pt] University of Washington Amazon FAR · 2026
Read TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning by Matthew M. Hong, Jesse Zhang, Anusha Nagabandi, Abhishek Gupta, [4pt] University of Washington Amazon FAR (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12236.txt.
Introduction
Matthew Hong, Jesse Zhang, Anusha Nagabandi, and Abhishek Gupta bridge cloning pretraining and reinforcement tuning for robot policies. Cloning yields narrow action spreads that starve downstream search. Their CSP method injects forward noise into policy inputs, spanning precise copying to broad range. Fine-tuning then uses TMRL (Timestep-Modulated Reinforcement Learning), which adjusts this setting on the fly by shifting the step, giving direct search control. The method lifts tuning speed and enables real-world tuning on hard handling tasks in less than an hour.
Catalog record
| CREATOR | Matthew M. Hong, Jesse Zhang, Anusha Nagabandi, Abhishek Gupta, [4pt] University of Washington Amazon FAR |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68384 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12236
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning Matthew M. Hong, Jesse Zhang, Anusha Nagabandi, Abhishek Gupta, [4pt] University of Washington Amazon FAR Abstract Fine-tuning pre-trained robot policies with reinforcement learning (RL) often inherits the bottlenecks introduced by pre-training with behavioral cloning (BC), which produces narrow action distributions that lack the coverage necessary for downstream exploration. We present a unified framework that enables the exploration necessary to enable efficient robot-policy finetuning by bridging BC pre-training and RL fine-tuning. Our pre-training method, Context-Smoothed Pre-training (CSP), injects forward-diffusion noise into policy inputs, creating a continuum between precise imitation and broad action coverage. We then fine-tune pre-trained policies via Timestep-Modulated Reinforcement Learning (TMRL), which trains the agent to dynamically adjust this conditioning during fine-tuning by modulating the diffusion timestep, granting explicit control over exploration. Integrating seamlessly with arbitrary policy inputs, e.g., states, 3D point clouds, or image-based VLA policies, we show that TMRL improves RL fine-tuning sample efficiency. Notably, TMRL enables successful real-world fine-tuning on complex manipulation tasks in under one hour. Videos and code available at https://weirdlabuw.github.io/tmrl/ . I Introduction A dominant paradigm for training real-world robotic policies is to first pre-train on large-scale demonstration datasets via imitation learning, and then fine-tune the resulting policy with reinforcement learning (RL) in deployment environments [ 50 , 51 , 21 , 61 , 60 , 29 , 58 , 19 , 55 , 54 ] . This RL fine-tuning stage is critical for improving task precision, throughput and robustness [ 24 , 29 , 26 ] , yet remains bottlenecked by sample efficiency due to the cost of real-world interaction. While prior work in robotics primarily focuses on improving the RL algorithms themselves, comparatively little attention has been paid to ensuring that pre-trained policies provide effective initializations for downstream RL. In this work, we propose a combined robot policy pre-training and RL fine-tuning framework that enables exploration for sample-efficient adaptation. Standard behavior cloning (BC) trains a policy to imitate demonstrator actions [ 38 , 2 ] . When demonstrations densely cover a context c (e.g., observation and task instruction), BC accurately models the conditional action distribution p(a\mid c) [ 13 , 9 ] . However, in sparsely covered regions and under distribution shift, BC overfits to observed data: the conditional support collapses and BC can assign near-zero probability to optimal actions. As a result, online rollouts during downstream adaptation yield little reward, causing reinforcement learning to fail to receive the learning signal needed to improve. Building on this insight, recent work has examined the relationship between pre-training via supervised learning and RL fine-tuning [ 46 , 7 , 56 , 50 , 6 ] , primarily in language domains. In robotics specifically, 50 shows that achieving high action coverage —the ability to sample actions a demonstrator would take—is necessary for efficient RL fine-tuning. Thus, they inject Gaussian action noise during training to increase coverage. However, there is a fundamental trade-off: random actions guarantee coverage but lack task proficiency, while BC policies can achieve high proficiency but have limited out-of-distribution coverage. A naive solution is to collect more data, but doing so for every new setting can be costly. Instead, we ask: how can we pre-train policies that adaptively broaden their action distribution during RL? Our key insight is to train policies that interpolate between the learned conditional p(a\mid c) and marginal p(a) action distributions. While p(a\mid c) offers precise behavior in familiar contexts, p(a) provides broad coverage needed for exploration. Interpolating between these extremes enables balancing exploitation and exploration based on the context’s novelty. We instantiate this idea through Context-Smoothed Policies (CSPs) , a pre-training framework that expands the support of BC policies by injecting noise into context inputs via a forward diffusion process . Introducing low levels of diffusion noise can alias similar contexts c [ 20 ] . Aliasing the input contexts leads to learning a broader conditional action distribution, since actions are “borrowed” across nearby contexts. As the noise increases, the policy smoothly transitions from a narrow conditional distribution p(a\mid c) toward a broad marginal distribution p(a) . At maximum noise, c contains no information and the policy recovers p(a) [ 63 ] , ensuring full training distribution coverage. We pre-train CSPs across all noise levels, enabling smooth interpolation between these extremes. Therefore, the tradeoff between action coverage and conditioning fidelity can be selected at inference time. Finally, having established a pre-training framework that exposes this tradeoff, the central challenge becomes how best to harness this flexible initialization during downstream RL finetuning. During RL fine-tuning, the optimal amount of context conditioning necessary—and therefore the balance between broad exploration and precise execution—can vary dramatically even within a single trajectory. To fully exploit our pre-trained CSPs, we introduce Timestep-Modulated Reinforcement Learning ( TMRL ) to dynamically adjust the diffusion timestep during RL. This mechanism provides an RL agent with an explicit control variable that modulates conditioning strength, enabling it to interpolate between conditional and marginal behaviors in the pre-trained policy and thereby explore more effectively. Ideally, a policy learns when to rely on precise imitation and when to broaden its action support. In practice, TMRL is simple and easy to implement, enabling steering/fine-tuning of diverse policies, from state-input diffusion policies to vision-language-action (VLA) models. We evaluate our proposed methodology across a range of simulated and real-world robotic tasks. First, we show that CSPs alone substantially improve action coverage and achieve stronger zero-shot performance on unseen tasks over standard BC and prior pre-training approaches. TMRL then converts this coverage advantage into significantly better RL sample efficiency on manipulation and navigation tasks in simulation, outperforming state-of-the-art steering methods even when the base policy contains sparse behavioral coverage. We further show that context-smoothing extends naturally to VLA and 3D-input policies, where noising VLA embeddings and point clouds, respectively, enables broader exploration across image-input and dexterous manipulation tasks. Finally, we demonstrate that our approach scales to the real world, enabling rapid RL of manipulation behaviors within an hour of experiment time while the baseline achieves near-zero success. II Related Work Existing research on RL fine-tuning in robotics spans a wide spectrum of topics. In this section, we review these paradigms and situate both context-smoothed policies and TMRL within them. Specifically, we differentiate our approach in (i) the design of pre-training objectives for action coverage, (ii) the explicit, dynamic tuning of exploration optimism during post-training, and (iii) the role of restricted information in inducing structured state aliasing. Pre-training for RL Fine-tuning. Standard practice in robot learning involves pre-training policies on large-scale demonstration datasets via offline RL or BC, followed by RL fine-tuning [ 50 , 51 , 21 , 61 , 60 , 29 , 58 , 19 , 55 , 54 , 24 , 29 , 26 ] . However, these imitation objectives learn the narrow conditional action distribution, p(a\mid c) . In low-density or out-of-distribution contexts, this conditional distribution overcommits to observed data, collapsing its support and assigning near-zero probability to potentially optimal actions. Recent findings in language modeling suggest that standard imitation losses are a poor predictor of downstream fine-tuning performance [ 46 , 7 , 56 , 6 ] . Instead, PostBC [ 50 ] demonstrates that successful RL fine-tuning of robot