Online Finetuning Decision Transformers with Pure RL Gradients
Junkai LuoUniversity of California, Riverside [email protected], Yinglun ZhuUniversity of California, Riverside [email protected] · 2026
Read Online Finetuning Decision Transformers with Pure RL Gradients by Junkai LuoUniversity of California, Riverside [email protected], Yinglun ZhuUniversity of California, Riverside [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00167.txt.
Description
Junkai Luo and coauthor Yinglun Zhu retrain Decision Transformers with pure reinforcement learning signals. Decision Transformers treat offline learning as sequence modeling. They work well before deployment, but online tuning is less explored. Most online versions still lean on supervised goals. The authors find a clash in one standard step.
Hindsight return relabeling helps supervised learning a lot. Yet it clashes with importance sampling methods like GRPO. That clash makes RL based tuning unstable. Their fix adapts GRPO to the transformer setting. Sub trajectory training sharpens credit assignment. Sequence level goals add stability and speed. Active sampling pushes search into unsure regions. Wide tests beat earlier online baselines. The results set new best scores on several benchmarks. Pure RL tuning, done right, can lift a sequence model further.
Catalog record
| CREATOR | Junkai LuoUniversity of California, Riverside [email protected], Yinglun ZhuUniversity of California, Riverside [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-00167