From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning

Pusen Dong, Tianchen Zhu, Yue Qiu, Haoyi Zhou, Jianxin Li · 2024

Read From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning by Pusen Dong, Tianchen Zhu, Yue Qiu, Haoyi Zhou, Jianxin Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08920.txt.

Introduction

Dong and co-authors let a learning agent read its safety rules as plain text instead of hand-built cost formulas, in a 2024 paper. Safe reinforcement learning demands that an agent finish tasks without breaking constraints, but writing a cost function for each rule takes expert effort. Their translator module reads both the written constraint and the agent's trajectory, and produces the training signal itself.

Policies trained this way broke rules less often than those trained with standard cost functions. Further tests show the translator can carry its understanding to new environments where the constraints have shifted, without retraining. The approach aims to make safety rules easier to state and reuse.

Catalog record

CREATORPusen Dong, Tianchen Zhu, Yue Qiu, Haoyi Zhou, Jianxin Li
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08920

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning Pusen Dong, Tianchen Zhu, Yue Qiu, Haoyi Zhou, Jianxin Li Abstract Safe reinforcement learning (RL) requires the agent to finish a given task while obeying specific constraints. Giving constraints in natural language form has great potential for practical scenarios due to its flexible transfer capability and accessibility. Previous safe RL methods with natural language constraints typically need to design cost functions manually for each constraint, which requires domain expertise and lacks flexibility. In this paper, we harness the dual role of text in this task, using it not only to provide constraint but also as a training signal. We introduce the Trajectory-level Textual Constraints Translator (TTCT) to replace the manually designed cost function. Our empirical results demonstrate that TTCT effectively comprehends textual constraint and trajectory, and the policies trained by TTCT can achieve a lower violation rate than the standard cost function. Extra studies are conducted to demonstrate that the TTCT has zero-shot transfer capability to adapt to constraint-shift environments. 1 Introduction In recent years, reinforcement learning (RL) has achieved remarkable success in multiple domains, such as go game [ 2 , 4 ] and robotic control [ 44 , 45 ] . However, deploying RL in real-world scenarios still remains challenging. Many real-world decision-making applications, such as autonomous driving [ 3 , 46 ] require agents to obey certain constraints while achieving the desired goals. To learn a safe constrained policy, some safe RL works [ 1 , 6 , 7 , 9 , 8 , 11 , 10 ] have proposed methods to maximize the reward while minimizing the constraint violations after training or during training. However, several limitations prevent the existing safe RL methods’ widespread use in real-world applications. Firstly, these methods often require mathematical or logical definitions of cost functions, which require domain expertise ( Limitation 1 ). Secondly, their cost function definitions are frequently specific to a particular context and cannot be easily generalized to new tasks with similar constraints ( Limitation 2 ). Lastly, most current safe RL methods focus on constraints that are logically simple, typically involving only one single entity or one single state [ 27 ] , which can’t represent the real-world safety requirements and lack universality ( Limitation 3 ). Table 1: Comparison of trajectory-level constraints and previous single state/entity constraints. Universal Trajectory-level constraints can model any constraint requirements in real-world scenarios. But single state/entity constraint can only model the constraint requirements on individual state/entity. Single state/entity constraint (previous) Trajectory-level constraint (ours) Don’t drive car. (single entity) Don’t drive car after you drink wine . (multi entities) Don’t touch lava. (single state) Don’t touch lava more that three times . (multi states) Don’t step in the gassed area. (single state) Don’t stay in the gas area for more than 5 minutes , your gas mask will fail. (multi states) Using natural language to provide constraints [ 14 , 13 , 12 ] is a promising approach to overcome Limitation 1 and 2 because natural language allows for flexible, high-level expression of constraints that can easily adapt to different scenarios. Regarding Limitation 3 , previous approaches primarily employ what we call the single state/entity textual constraint . The single-state/entity textual constraint focuses solely on constraints related to one specific state or entity, limiting the ability to model complex safety requirements in real-world scenarios. Many safety requirements involve interactions and dependencies among multiple states or entities over time. By only addressing a single state or entity, these constraints fail to capture the dynamic relationships and temporal aspects that are crucial for ensuring safety in complex environments. So we suggest using a more generalizable constraint type trajectory-level textual constraint . The trajectory-level textual constraint is a more universal constraint with complex logic semantics involving multiple states/entities. It is a nonlinear combination of multiple entities or multiple environment states and can model any constraint requirements in real-world scenarios. The trajectory-level constraints in natural language form are the highly abstract expression of the agent’s behavior guidelines, serving as a more natural and straightforward way to introduce constraints. Notably, the set of trajectory-level constraints encompasses single-state constraints, as any single-state constraint can be reformulated as an equivalent trajectory-level constraint. Examples of trajectory-level textual constraint and single state/entity textual constraint are presented in Table 1 . Employing trajectory-level textual constraints across the entire trajectory poses two significant challenges. Firstly, determining whether an RL agent violates textual constraint over a trajectory is non-trivial, it needs to have the perception of the historical states and actions observed along the trajectory ( Challenge 1 ). Secondly, the trajectory-level safety problem is susceptible to sparse cost [ 15 ] , where cost is only imposed when the agent violates textual constraints at the final time step, making it challenging for the agent to learn which actions contribute to a gradual escalation of risk ( Challenge 2 ). For instance, the agent needs to learn constraints like, "Don’t touch lava after you touch grass." Without intermediate feedback, however, it struggles to understand how early actions, such as stepping on grass, contribute to eventual violations. To address Challenge 1 , we propose a general approach to align the trajectory’s factual logic with the text’s semantic logic, eliminating the need for manual encoding or separate models for each type of constraint. Our method employs a sequence model [ 16 , 47 ] for modeling agent historical interactions with the environment, and a pre-trained language model (LM) [ 18 ] to comprehend natural language constraints. We then maximize the embedding similarities between matching pairs (trajectory, text) and minimize the embedding similarities between non-matching pairs using a contrastive learning approach [ 19 ] similar to CLIP [ 20 ] . By calculating the similarity between the textual embeddings of the constraint and the trajectory, we can predict whether a constraint is violated in this trajectory. Our method uniquely leverages text as both a source of constraints and a unified supervisory signal for trajectory encoding. In this dual role, text not only provides constraints but also guides the training process, enabling the model to naturally handle diverse semantic constraints without requiring specific model adjustments for each type. This design allows for a more flexible and generalizable system, significantly simplifying the handling of complex, multi-dimensional constraints. In addition, to address the issue of cost sparsity ( Challenge 2 ), we introduce a method for temporal credit assignment [ 21 ] . The proposed approach involves decomposing the one episodic cost of the textual constraint into multiple parts and allocating them to each state-action pair within the trajectory. This method offers denser cost signals regarding the relationship between the textual constraint and the agent’s every action. It informs the agent which behaviors are risky and which are safer, thereby enhancing safety and aiding model performance. Our experiments demonstrate that the proposed method can effectively address Challenge 1 and Challenge 2 . In both 3D navigation [ 37 ] and 2D grid exploration [ 14 ] tasks, agents trained using our method achieve significantly lower violation rates (up to 4.0x) compared to agents trained with ground-truth cost functions while maintaining comparable rewards and more importantly, our

Read the complete text