STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order

Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen · 2026

Read STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order by Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08107.txt.

Introduction

Chengyang Gu, Yuxin Pan, Hui Xiong, and Yize Chen improve offline reinforcement learning under sparse rewards. Their 2026 arXiv paper notes that offline learning trains policies from fixed datasets, avoiding risky online trials. Long tasks with rare rewards still defeat many methods. Goal-based and hierarchical methods split tasks and add intermediate rewards, but they often ignore time order among subgoals. They also lean on rough reward shaping, which yields weak policies. STO-RL uses language model guidance for subgoal time order to address both gaps.

Catalog record

CREATORChengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08107

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen Abstract . Offline reinforcement learning (RL) enables policy learning from pre-collected datasets, avoiding costly and risky online interactions, but it often struggles with long-horizon tasks involving sparse rewards. Existing goal-conditioned and hierarchical offline RL methods decompose such tasks and generate intermediate rewards to mitigate limitations of traditional offline RL, but usually overlook temporal dependencies among subgoals and rely on imprecise reward shaping, leading to suboptimal policies. To address these issues, we propose STO-RL (Offline RL using LLM-Guided Subgoal Temporal Order), an offline RL framework that leverages large language models (LLMs) to generate temporally ordered subgoal sequences and corresponding state-to-subgoal-stage mappings. Using this temporal structure, STO-RL applies potential-based reward shaping to transform sparse terminal rewards into dense, temporally consistent signals, promoting subgoal progress while avoiding suboptimal solutions. The resulting augmented dataset with shaped rewards enables efficient offline training of high-performing policies. Evaluations on four discrete and continuous sparse-reward benchmarks demonstrate that STO-RL consistently outperforms state-of-the-art offline goal-conditioned and hierarchical RL baselines, achieving faster convergence, higher success rates, and shorter trajectories. Ablation studies further confirm STO-RL’s robustness to imperfect or noisy LLM-generated subgoal sequences, demonstrating that LLM-guided subgoal temporal structures combined with theoretically grounded reward shaping provide a practical and scalable solution for long-horizon offline RL. 1. Introduction In recent years, Deep Reinforcement Learning (DRL) has emerged as a powerful paradigm for solving sequential decision-making problems, achieving remarkable success in domains ranging from game-playing ( Goldwaser and Thielscher 2020 ) to robotics ( Nguyen and La 2019 ) . Despite these achievements, DRL methods usually require extensive online interaction with the environment, which can be costly, risky, or infeasible in real-world applications such as autonomous driving or medical treatment planning ( Isele et al. 2018 ; Yauney and Shah 2018 ) . To overcome this limitation, offline reinforcement learning (offline RL) ( Kumar et al. 2020 ) has been investigated, where agents learn policies entirely from pre-collected datasets without further interactions with targeting environments. However, to date offline RL faces significant challenges in long-horizon, goal-reaching tasks with sparse rewards ( Wu et al. 2024 ) , which are practical when agents interacting with the real world. In these settings, rewards are often provided only at the final state, and in limited offline datasets, the lack of intermediate feedback also hinders efficient reward propagation. These make RL agents very challenging to learn effective behaviors ( Holmes and Chi 2025 ; Yunis et al. 2024 ) . To mitigate this issue, goal-conditioned offline RL extends traditional offline RL by conditioning policies or value functions on goal representations, enabling more flexible and generalizable goal-directed behaviors ( Ma et al. 2022 ; Park et al. 2024 ; Ding et al. 2019 ) . Complementarily, hierarchical reinforcement learning (HRL) has been proposed to tackle sparse-reward and long-horizon problems by introducing a multi-level decision-making hierarchy ( Schwab and Ray 2017 ; Nachum et al. 2018 ; Nachum et al. 2019 ; Osa et al. 2019 ) . In such a framework, a high-level policy generates subgoals to guide long-term planning, while a low-level policy learns to achieve these subgoals using primitive actions. By decomposing complex tasks into sequences of subgoal-conditioned subtasks, HRL can improve learning efficiency and exploration, but at an additional cost of algorithm design and complexity ( Zhang et al. 2021 ) . Figure 1 . Overview of STO-RL. The agent first prompts an LLM to decompose task instructions into temporally ordered subgoals \mathcal{G}=\{G_{1},G_{2},...,G_{K}\} with a mapping function h linking each state s_{t} to its subgoal index k_{t} ; then applies potential-based reward shaping to assign dense rewards to each transition (s_{t},a_{t},s_{t+1}) in dataset D based on the mapping function h , followed by training an offline RL agent on the augmented dataset to learn the optimal policy. Despite these advances, both offline goal-conditioned RL and offline hierarchical RL continue to face two fundamental challenges: (1) Learning Effective Subgoals ( Mesbah et al. 2024 ; Kim et al. 2021 ; Chane-Sane et al. 2021 ) . Generating informative and achievable subgoals is critical for their success, because high-level and goal-conditioned policies rely on them to guide low-level actions toward long-horizon goals, directly influencing learning efficiency and task success. However, discovering such subgoals remains difficult due to limited data coverage, sparse rewards, and the presence of suboptimal trajectories and noisy transitions in offline datasets ( Shin and Kim 2023 ; Park et al. 2023 ; Wang et al. 2023 ; Kujanpää et al. 2023 ; Rosete-Beas et al. 2023 ) . Furthermore, existing approaches often fail to model the temporal dependencies among subgoals ( Baek et al. 2025 ; Xu and Fekri 2024 ; Ahn et al. 2025 ) , making it challenging to learn temporally coherent subgoal sequences. This limitation leads high-level policies to generate subgoals that are inconsistent with task progression and hinder effective hierarchical coordination. (2) Reward Shaping ( Mezghani et al. 2023 ; Gao et al. 2024 ; Okudo and Yamada 2021 ) . To alleviate the challenge of sparse rewards and enhance learning efficiency, reward shaping is commonly employed to provide denser feedback signals that guide agents toward meaningful, intermediate progress. While shaping schemes such as distance-based potential functions can introduce unintended local optima and misguide policy learning, ultimately leading to suboptimal performance ( Liu et al. 2022 ) . In this work, we identify large language model (LLM) as the critical aid, and LLM-based planning can redefine how subgoals and rewards are discovered in long-horizon, sparse-reward problems. By leveraging their powerful natural language understanding and reasoning capabilities, LLMs can decompose complex tasks into structured, temporally ordered subgoal sequences, enabling zero-shot or few-shot plan generation ( Yao et al. 2023 ; Saha et al. 2024 ; Shukla et al. 2023 ) . Interestingly, recent works demonstrate that LLMs can also be applied to design effective and dense reward functions for goal-reaching tasks ( Xie et al. 2023 ; Pouplin et al. 2024 ; Bhambri et al. 2024 ) . However, few studies jointly address both perspectives: LLM-based high-level planners mostly rely on pre-trained low-level agents to execute subgoals, while LLM-based reward-shaping mechanisms also often neglect the temporal dependencies among subgoals, leading to inconsistencies and reduced reliability in hierarchical decision-making. Motivated by these promises and gaps between LLM planners and reward shaping, in this paper we propose Offline R einforcement L earning using LLM-Guided S ub-goal T emporal O rder (STO-RL) , a unified framework designed to simultaneously achieve subgoal discovery and reward shaping in the offline setting. Specifically, STO-RL prompts an LLM with task instructions and environment context to generate a sequence of temporally ordered subgoals, along with a mapping function that assigns each state to its corresponding subgoal stage. LLM excels at interpreting and generalizing these goal-reaching abstractions and relations. Leveraging this temporal structure, STO-RL applies a potential-based reward shaping scheme that integrates subgoal order into the reward function, effectively transforming sparse terminal rewards into dense signals that promotes subgoal progress to the final goal while theoretically avoiding suboptimal shaping. The resulting augmented dataset with shaped rewards enables efficient offline training of an optimal policy from a limited offline dataset. To validate

Read the complete text