Online Finetuning Decision Transformers with Pure RL Gradients

Junkai LuoUniversity of California, Riverside [email protected], Yinglun ZhuUniversity of California, Riverside [email protected] · 2026

Read Online Finetuning Decision Transformers with Pure RL Gradients by Junkai LuoUniversity of California, Riverside [email protected], Yinglun ZhuUniversity of California, Riverside [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00167.txt.

Description

Junkai Luo and coauthor Yinglun Zhu retrain Decision Transformers with pure reinforcement learning signals. Decision Transformers treat offline learning as sequence modeling. They work well before deployment, but online tuning is less explored. Most online versions still lean on supervised goals. The authors find a clash in one standard step.

Hindsight return relabeling helps supervised learning a lot. Yet it clashes with importance sampling methods like GRPO. That clash makes RL based tuning unstable. Their fix adapts GRPO to the transformer setting. Sub trajectory training sharpens credit assignment. Sequence level goals add stability and speed. Active sampling pushes search into unsure regions. Wide tests beat earlier online baselines. The results set new best scores on several benchmarks. Pure RL tuning, done right, can lift a sequence model further.

Catalog record

CREATORJunkai LuoUniversity of California, Riverside [email protected], Yinglun ZhuUniversity of California, Riverside [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00167

Browse the full library · About the Tower of Babel