One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning
Zijian Guo, İlker Işık, H. M. Sabbir Ahmad, Wenchao Li · 2025
Read One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning by Zijian Guo, İlker Işık, H. M. Sabbir Ahmad, Wenchao Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01561.txt.
Introduction
Guo and co-authors present GenZ-LTL, a method that lets learning agents handle new task rules stated in formal logic, in a research paper from 2025. The rules, written in linear temporal logic, can nest long goals and safety limits. Older methods struggle with such nesting and cannot tell when a subgoal is impossible.
The method splits a rule into a chain of reach-and-avoid subgoals using automaton structure. It then solves one subgoal at a time with safe learning formulations. A state-trimming trick curbs the blowup of subgoal combinations. Tests show the method beats rivals at generalizing to unseen rules without retraining.
Catalog record
| CREATOR | Zijian Guo, İlker Işık, H. M. Sabbir Ahmad, Wenchao Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning Zijian Guo, İlker Işık, H. M. Sabbir Ahmad, Wenchao Li Abstract Generalizing to complex and temporally extended task objectives and safety constraints remains a critical challenge in reinforcement learning (RL). Linear temporal logic (LTL) offers a unified formalism to specify such requirements, yet existing methods are limited in their abilities to handle nested long-horizon tasks and safety constraints, and cannot identify situations when a subgoal is not satisfiable and an alternative should be sought. In this paper, we introduce GenZ-LTL, a method that enables zero-shot generalization to arbitrary LTL specifications. GenZ-LTL leverages the structure of Büchi automata to decompose an LTL task specification into sequences of reach-avoid subgoals. Contrary to the current state-of-the-art method that conditions on subgoal sequences, we show that it is more effective to achieve zero-shot generalization by solving these reach-avoid problems one subgoal at a time through proper safe RL formulations. In addition, we introduce a novel subgoal-induced observation reduction technique that can mitigate the exponential complexity of subgoal-state combinations under realistic assumptions. Empirical results show that GenZ-LTL substantially outperforms existing methods in zero-shot generalization to unseen LTL specifications. 1 Introduction Generalization is a critical aspect of reinforcement learning (RL), which aims to develop policies that are capable of adapting to novel and unseen tasks oh2017zero ; taiga2023investigating ; hendawy2024multitask and varying safety constraints cho2024constrained ; guo2025constraintconditioned . Various methods have been proposed and show promising results in applications such as autonomous driving tang2024efficient ; ma2024unsupervised , robotics kim2023trust ; lin2023safe ; kim2023learning , and healthcare cui2024automated ; rafiei2024meta . However, most existing approaches fall short in handling complex and temporally extended task objectives and safety constraints. For example, an autonomous vehicle may be instructed to reach a sequence of destinations for passenger pick-up and drop-off while adhering to traffic rules such as speed limits and yielding at intersections. Temporal logic provides a principled framework for specifying system behaviors over time. Linear temporal logic (LTL) pnueli1977temporal , in particular, has been considered extensively in single-task and multi-task RL settings hasanbeig2018LogicallyConstrained ; GLeon2020SystematicGT ; voloshin2023eventual . LTL has well-defined semantics and a compositional syntax, which facilitates the unambiguous specification of task objectives and safety constraints. However, existing methods often fail to account for the potential conflicting objectives within a specification vaezipoor2021ltl2action ; qiu2023instructing ; jackermeierdeepltl . For example, an autonomous vehicle navigating between multiple destinations aims to minimize travel time, but adhering to speed limits and other safety constraints may slow the vehicle down. In such cases, the vehicle must prioritize satisfying safety requirements over optimizing for travel time. In this paper, we propose a novel framework for learning RL policies that generalize in a zero-shot manner to satisfy arbitrary LTL specifications, including both finite-horizon and infinite-horizon tasks. The core idea is to exploit the structure of the equivalent Büchi automata to decompose an LTL specification into individual subgoals, each comprising a reach component that defines task progression and an avoid component that encodes safety constraints. A goal-conditioned policy is then trained to achieve the subgoals one at a time through a safe RL formulation that can effectively handle the safety constraints and optimize for efficient satisfaction of the subgoal. During training, we sample different subgoals from the space of possible subgoals to train the goal-conditioned policy. In addition, we introduce a novel subgoal-induced observation reduction technique to address the exponential complexity of observation-subgoal combinations. This technique exploits symmetry in the state observations and extracts information only relevant to the current subgoals. At test time, given an arbitrary LTL specification, we use its equivalent Büchi automaton as well as the learned value function to guide the subgoal selection. Furthermore, we introduce a timeout-based subgoal-switching mechanism that tracks subgoal satisfaction and enables the switching to alternative subgoals if the current one turns out to be unsatisfiable, a situation that existing methods fail to manage. Our main contributions are summarized as follows. • We propose a novel approach for learning RL policies that can satisfy arbitrary LTL specifications in a zero-shot manner by sequentially completing individual subgoals. To our knowledge, this is the first safe RL formulation with state-wise constraints for LTL task satisfaction. • We introduce a subgoal-induced observation reduction technique to mitigate the combinatorial explosion of observation-subgoal pairs and enable efficient generalization, along with a timeout mechanism that handles subgoal switching and unsatisfiable subgoals. • We conduct comprehensive experiments across a set of navigation-style environments and diverse task specifications, demonstrating that our method consistently outperforms state-of-the-art baselines in zero-shot generalization to arbitrary LTL specifications. 2 Related Work Specification-guided and goal-conditioned RL. Various methods have been proposed to guide RL agents by extending traditional goal-conditioned RL ( liu2022goal ) to temporal-logic goals. Early works focused on learning policies for a single and fixed specification, including methods based on LTL specifications and automaton-like models ( hasanbeig2018LogicallyConstrained ; hahn2023Mungojerrie ; voloshin2023eventual ; shao2023Sample ; shah2024LTLConstrained ; bagatella2024Directed ) , and methods leveraging quantitative semantics of specifications ( li2017reinforcement ; jothimurugan2019composable ; shah2024deep ; guo2024temporal ) . More recently, the importance of adapting to changing conditions in real-world applications has brought increased attention to zero-shot generalization to new specifications. Existing approaches can be broadly categorized into subgoal decomposition ( leon2021Systematic ; leon2022Nutshell ; liu2024Skill ; qiu2023instructing ; jackermeierdeepltl ) and direct specification/automata encoding ( vaezipoor2021ltl2action ; yalcinkaya2024compositional ; yalcinkaya2025provably ) . Our work follows the direction of subgoal decomposition, but shows that it is more effective to solve the subgoals one at a time through proper safe RL formulations and subgoal-induced observation reduction. RL with safety constraints. Various methods have been proposed to solve RL problems with safety constraints, such as policy optimization-based methods zhang2020first ; Yang2020Projection-Based ; jayant2022model ; xiong2023provably , Gaussian processes-based approaches ( wachi2018safe ; zhao2023probabilistic ) , and control theory-based methods, including Hamilton-Jacobian (HJ) reachability fisac2019bridging ; yu2022reachability ; so2023solving ; ganai2023iterative ; zheng2024safe and control barrier functions ( gangopadhyay2022safe ; cheng2023safe ) . We point the readers to gu2022review for a comprehensive survey of safe RL techniques. In the context of formal specification-guided RL, the interplay between task progression and safety constraints remains underexplored. Most existing methods simply assign a positive reward when a specification or a subgoal is satisfied, and a negative reward at episode termination if unsatisfied. This approach is ineffective in achieving safety ( achiam2017constrained ; dalal2018safe ; bharadhwaj2020conservative ) , especially when the task progression objective and the safety requirements exert conflicting influences on policy learning. In this paper, we show how to prioritize safety satisfaction properly by incorporating Hamilton-Jacobi reachability constraints. 3 Preliminaries Reinforcement learning. RL agents learn policies by interacting with environments, which is usually modeled as a Markov Decision Process (MDP), defined by the tuple \mathcal{M}:=(\mathcal{S},\mathcal{A},P,r,\gamma,d_{0}) , where \mathcal{S} is the state space and \mathcal{A} is the action space, P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto[0,1] defines the transition dynamics, r:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R} is the reward function, \gamma\in[0,1) is the discount factor, and d_{0}\in\Delta(\mathcal{S}) is the initial state distribution. Let \pi:\mathcal{S}\times\mathcal{A}\mapsto[0,1] denote a policy and \tau=\{s_{t},a_{t},r_{t}\}_{t=0}^{\infty} denote a trajectory where r_{t}=r(s_{t},a_{t}) . The value function under policy \pi is V^{\pi}_{r}(s)=\mathbb{E}_{\tau\sim\pi,s_{0}=s}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}] and the corresponding state-action value function is Q_{r}^{\pi}(s,a)=\mathbb{E}_{\tau\sim\pi,s_{0}=s,a_{0}=a}[\sum_{t=1}^{\infty}\gamma^{t}r_{t}] . An additional constraint violation function h:\mathcal{S}\mapsto\mathbb{R} can be used to model