Online Finetuning Decision Transformers with Pure RL Gradients Junkai LuoUniversity of California, Riverside junkail@ucr.edu, Yinglun ZhuUniversity of California, Riverside yzhu@ucr.edu Abstract Decision Transformers (DTs) have emerged as a powerful framework for sequential decision making by formulating offline reinforcement learning (RL) as a sequence modeling problem. However, extending DTs to online settings with pure RL gradients remains largely unexplored, as existing approaches continue to rely heavily on supervised sequence-modeling objectives during online finetuning. We identify hindsight return relabeling—a standard component in online DTs—as a critical obstacle to RL-based finetuning: while beneficial for supervised learning, it is fundamentally incompatible with importance sampling-based RL algorithms such as GRPO, leading to unstable training. Building on this insight, we propose new algorithms that enable online finetuning of Decision Transformers using pure reinforcement learning gradients. We adapt GRPO to DTs and introduce several key modifications, including sub-trajectory optimization for improved credit assignment, sequence-level likelihood objectives for enhanced stability and efficiency, and active sampling to encourage exploration in uncertain regions. Through extensive experiments, we demonstrate that our methods outperform existing online DT baselines and achieve new state-of-the-art performance across multiple benchmarks, highlighting the effectiveness of pure-RL-based online finetuning for Decision Transformers. † † footnotetext: † Project lead and corresponding author. 1 Introduction The transformer architecture ( Vaswani et al. 2017 ) lies at the core of modern foundation models. Large language models (LLMs), in particular, have demonstrated remarkable generalization and reasoning capabilities through a simple yet powerful paradigm: large-scale pretraining followed by supervised and reinforcement learning (RL)-based finetuning ( Radford et al. 2018 ; Brown et al. 2020 ; Ouyang et al. 2022 ; Achiam et al. 2023 ; Comanici et al. 2025 ) . Inspired by this success, the Decision Transformer (DT, Chen et al. 2021 ) introduces transformers to sequential decision making by reframing classical RL as a conditional sequence modeling problem. As an offline RL method, DTs are trained with a supervised objective on pre-collected trajectories, effectively performing imitation learning ( Hussein et al. 2017 ) while conditioning on return-to-go (RTG) tokens. The Online Decision Transformer ( ODT , Zheng et al. 2022 ) extends DTs to the online setting by enabling finetuning after offline pretraining. During online finetuning, ODT collects new trajectories and applies hindsight return relabeling , replacing intended RTGs with realized returns to align trajectories with their outcomes, mirroring the offline training procedure. Subsequent work further augments ODT with TD3 ( Fujimoto et al. 2018 ) gradients, resulting in ODT + TD3 ( Yan et al. 2024 ) . Despite these advances, existing approaches to online finetuning of DTs remain dominated by supervised objectives: ODT relies exclusively on supervised loss, while ODT + TD3 assigns only a small weight to RL gradients. In contrast, recent progress in LLM finetuning shows that pure RL methods—such as Group Relative Policy Optimization (GRPO, Shao et al. 2024 )—can substantially improve reasoning and alignment ( Guo et al. 2025 ; Team 2025 ) . This contrast naturally raises an important question: Can Decision Transformers be finetuned online using pure RL gradients? To investigate this question, we revisit the training paradigm of existing online DT methods and identify a fundamental limitation. We show that hindsight return relabeling ( Zheng et al. 2022 ) , while beneficial for supervised learning, is fundamentally incompatible with on-policy RL algorithms that rely on importance sampling. Relabeling RTGs used during rollout with returns observed afterward introduces a mismatch in importance ratios, leading to unstable optimization and degraded performance ( Fig. 1 ). Removing hindsight return relabeling is therefore a necessary first step toward applying importance sampling-based algorithms, such as GRPO and PPO, to online finetuning of Decision Transformers. Building on this insight, we develop a new framework for online finetuning of DTs using pure RL gradients . Specifically, we adapt GRPO to Decision Transformers (GRPO - DT) and introduce three key modifications: (i) a sub-trajectory-based optimization objective that enables fine-grained credit assignment, supported by either environment resetting ( Mhammedi et al. 2024 ; Kazemnejad et al. 2025 ) or an auxiliary Q-function; (ii) a sequence-level importance ratio that improves training stability and efficiency; and (iii) an active sampling strategy that prioritizes uncertain states to enhance exploration. Together, these components enable RL-only finetuning of pretrained DTs and yield new state-of-the-art performance across multiple benchmarks. In addition, we adapt Proximal Policy Optimization (PPO, Schulman et al. 2017 ) to Decision Transformers (PPO - DT), achieving competitive results where prior PPO-based approaches failed ( Yan et al. 2024 ) . Contributions. Our main contributions are summarized as follows: (i) We identify hindsight return relabeling—a standard component in existing online DT methods—as a critical obstacle to reinforcement learning-based finetuning: while effective for supervised objectives, it is incompatible with importance sampling-based policy gradient algorithms such as GRPO and PPO. (ii) We propose GRPO - DT, an adaptation of GRPO for Decision Transformers that enables pure-RL online finetuning by integrating sub-trajectory optimization for improved credit assignment, sequence-level importance ratios for enhanced stability and efficiency, and active state sampling for better exploration. These modifications are general and may be of independent interest beyond DTs. (iii) We conduct extensive experiments and demonstrate that pure RL finetuning—via GRPO - DT and PPO - DT—outperforms existing online DT baselines and achieves new state-of-the-art performance across multiple benchmarks. Paper organization. The remainder of the paper is organized as follows. Section 2 reviews background and preliminaries. Section 3 presents our methods. Section 4 reports experimental results. Section 5 discusses related work, and Section 6 concludes the paper. 2 Preliminaries Markov Decision Process. We formulate the reinforcement learning environment as a Markov Decision Process (MDP), defined by a tuple \mathcal{M}=(\mathcal{S},\mathcal{A},P,R) . Here, \mathcal{S} is the state space, \mathcal{A} is the action space, P(s_{h+1}\mid s_{h},a_{h}) is the probability transition function, R(s_{h},a_{h}) is the reward function. At each timestep h=1,\dots,H , the agent observes a state s_{h}\in\mathcal{S} , selects an action a_{h}\in\mathcal{A} according to a policy \pi(a_{h}\mid s_{h}) , transitions to the next state s_{h+1}\sim P(\cdot\mid s_{h},a_{h}) , and receives a reward r_{h}=R(s_{h},a_{h}) . The goal is to learn a policy \pi that maximizes the expected cumulative reward \mathbb{E}_{\mathcal{M},\pi}\!\left[\sum_{h=1}^{H}r_{h}\right] . Decision Transformer (DT). Decision Transformer ( Chen et al. 2021 ) represents a powerful paradigm for offline reinforcement learning, formulating decision making as a sequence modeling problem with pre-collected training trajectories. A DT trajectory consists of three types of tokens: return-to-go (RTG), state, and action, where the RTG g_{h}\vcentcolon=\sum_{\macc@depth\char 1\relax\macc@set@skewchar\macc@nested@a 111{h}=h}^{H}r_{\macc@depth\char 1\relax\macc@set@skewchar\macc@nested@a 111{h}} represents the sum of future reward from step h onward. DT leverages the transformer architecture ( Vaswani et al. 2017 ) to autoregressively learn a policy from pre-collected trajectories. DTs are trained on fixed-length trajectory segments ( Chen et al. 2021 ) : let m denotes the context length, the DT learns to generate the next action a_{h} based on past interactions ((g,s,a)_{h-m+1:h-1},g_{h},s_{h})\vcentcolon=(g_{h-m+1},s_{h-m+1},a_{h-m+1},\dots,g_{h},s_{h}) of context length m . The model is trained via supervised learning by minimizing the mean squared error (MSE) between the predicted action \pi(a_{h}\mid((g,s,a)_{h-m+1:h-1},g_{h},s_{h})) and the ground-truth action a_{h} . When the context is clear, we use the shorthand \pi(a_{h}\mid s_{h},g_{h})\vcentcolon=\pi(a_{h}\mid((g,s,a)_{h-m+1:h-1},g_{h},s_{h})) . During evaluation and deployment, the learner specifies a desired initial RTG, since the ground-truth future RTG isn’t known in advance, and leverages the DT to autoregressively generate the next action and interact with the environment. Online finetuning of Decision Transformers. Online Decision Transformer ( ODT , Zheng et al. 2022 ) extends offline DT to the online setting by first conducting offline pretraining and then online finetuning with interactively collected data. The offline pretraining stage largely follows the standard DT training procedure. In the online finetuning stage, the DT is deployed into the environment with a desired initial RTG g_{\mathsf{online}} to collect new trajectories that gradually replacing old trajectories stored in the replay buffer; the replay buffer is initialized with the offline trajectories. For online collected trajectories, ODT applies hindsight return relabeling ( Andrychowicz et al. 2017 ; Ghosh et al. 2019 ) to relabel the RTG tokens based on the actual achieved RTG g_{\mathsf{actual}} . ODT adopts a stochastic Gaussian policy to account for exploration in the online setting. However, similar to offline DT, ODT still learns via a supervised learning objective of minimizing the negative log-likelihood loss. While ODT improves model performance during online finetuning, recently, Yan et al. 2024 pointed out that ODT fails in settings with medium or low-quality offline data due to the sole use of the supervised learning objective. The supervised learning objective learns \frac{\partial a}{\partial\mathsf{RTG}} , i.e., how action changes as the target RTG varies, since DT models actions conditioned on RTGs. However, what actually drives online policy improvement is \frac{\partial\mathsf{RTG}}{\partial a} , i.e., how RTG responds to action adjustments, especially when offline pretraining data is not of high-quality; see section 3.1 in Yan et al. 2024 for more details. To enable better online improvements, Yan et al. 2024 propose ODT + TD3, which augments the supervised ODT objective (with hindsight return relabeling) with RL gradients from TD3 ( Fujimoto et al. 2018 ; Fujimoto and Gu 2021 ) to guide online exploration and adaptation. However, ODT + TD3 still prioritizes the supervised ODT loss, assigning only a small weight to the RL gradients during optimization. Group Relative Policy Optimization (GRPO). GRPO is a reinforcement learning algorithm initially proposed for large language models (LLMs) finetuning ( Shao et al. 2024 ; Guo et al. 2025 ) . It simplifies Proximal Policy Optimization (PPO, Schulman et al. 2017 ) by removing the need for a value model to estimate the advantages. Instead, GRPO samples multiple responses per question and uses their average reward as the baseline for advantage calculation. Specifically, for each query q\sim\Delta(Q) sampled from the question distribution \Delta(Q) , the model generates a group of G responses \{o_{1},\cdots,o_{G}\} based on policy \pi_{\theta_{\mathsf{old}}} . A reward r_{i} is computed for each response o_{i} , usually with the help of a reward model. GRPO optimizes the policy model by maximizing the following objective: \displaystyle J_{\mathsf{GRPO}}(\theta) \displaystyle=\mathbb{E}_{q\sim{\Delta(Q)},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\mathsf{old}}(\cdot\mid q)}} \displaystyle\quad\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{h=1}^{|o_{i}|}\min\left(w_{i,h}(\theta)\widehat{A}_{i},\,\mathsf{clip}\left(w_{i,h}(\theta),1-\varepsilon,1+\varepsilon\right)\widehat{A}_{i}\right)-\beta D_{\mathsf{KL}}\left(\pi_{\theta}\parallel\pi_{\theta_{\mathsf{ref}}}\right), (1) where \widehat{A}_{i}=\frac{r_{i}-\mathrm{mean}(\{r_{1},r_{2},\cdots,r_{G}\})}{\mathrm{std}(\{r_{1},r_{2},\cdots,r_{G}\})} denotes the advantage of the i -th rollout, w_{i,h}(\theta)=\frac{\pi_{\theta}(o_{i,h}|q,o_{i,