Accelerated Learning with Linear Temporal Logic using Differentiable Simulation

Alper Kamil Bozkurt, Calin Belta, Ming C. LinDepartment of Computer ScienceUniversity of Maryland, College Park {bozkurt,calin,lin}@umd.edu · 2025

Read Accelerated Learning with Linear Temporal Logic using Differentiable Simulation by Alper Kamil Bozkurt, Calin Belta, Ming C. LinDepartment of Computer ScienceUniversity of Maryland, College Park {bozkurt,calin,lin}@umd.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01167.txt.

Introduction

Alper Bozkurt, Calin Belta, and Ming Lin speed up robot learning from formal rules in a 2025 paper. Safety rules written in temporal logic say what must happen over a whole run, not just at one instant. Such rules give sparse feedback, so learning crawls, and hand-shaped rewards can break the rule's meaning.

Their framework pipes the rule's automaton into a differentiable simulator, softening its switches so gradients flow. Theory bounds the gap between the soft and exact objectives in both steady and random settings. On hard continuous control tasks, training sped up greatly and returns doubled over discrete baselines. The method also plugs into reward machines unchanged.

Catalog record

CREATORAlper Kamil Bozkurt, Calin Belta, Ming C. LinDepartment of Computer ScienceUniversity of Maryland, College Park {bozkurt,calin,lin}@umd.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01167

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerated Learning with Linear Temporal Logic using Differentiable Simulation Alper Kamil Bozkurt, Calin Belta, Ming C. LinDepartment of Computer ScienceUniversity of Maryland, College Park {bozkurt,calin,lin}@umd.edu Abstract To ensure learned controllers comply with safety and reliability requirements for reinforcement learning in real-world settings remains challenging. Traditional safety assurance approaches, such as state avoidance and constrained Markov decision processes, often inadequately capture trajectory requirements or may result in overly conservative behaviors. To address these limitations, recent studies advocate the use of formal specification languages such as linear temporal logic (LTL), enabling the derivation of correct-by-construction learning objectives from the specified requirements. However, the sparse rewards associated with LTL specifications make learning extremely difficult, whereas dense heuristic-based rewards risk compromising correctness. In this work, we propose the first method, to our knowledge, that integrates LTL with differentiable simulators, facilitating efficient gradient-based learning directly from LTL specifications by coupling with differentiable paradigms. Our approach introduces soft labeling to achieve differentiable rewards and states, effectively mitigating the sparse-reward issue intrinsic to LTL without compromising objective correctness. We validate the efficacy of our method through experiments, demonstrating significant improvements in both reward attainment and training time compared to the discrete methods. † † footnotetext: ∗ equal supervising 1 Introduction The growing demand for artificial intelligence (AI) systems to operate in a wide range of environments underscores the need for systems that can learn through interaction with their environments, without relying on human intervention. Reinforcement learning (RL) has emerged as a powerful tool for training controllers to perform effectively in uncertain settings with intricate, high-dimensional, and nonlinear dynamics. Recent advances in RL have enabled attainment of high-performance controllers in a variety of applications [ 1 ] , such as robotic arm control [ 2 ] , hand manipulation [ 3 ] , legged locomotion [ 4 , 5 ] , navigation in crowded spaces [ 6 ] , and robot-assisted surgery [ 7 ] . Despite the promising results in controlled environments, deploying learned controllers in real-world systems –where malfunctioning can be costly or hazardous– requires not only high performance but also strict compliance with formally specified safety and reliability requirements. Therefore, ensuring that learned controllers meet these critical specifications is essential to fully realize the potential of AI systems in real-world applications. Safety in learning is often modeled with constrained Markov decision processes (MDPs) [ 8 , 9 , 10 , 11 , 12 ] , where the accumulated cost must be within a budget. However, additive cost functions may not reflect real-world safety, as assigning meaningful costs to harms is challenging. Alternative approaches define safety by avoiding unsafe states or actions [ 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ] , which is simpler than designing cost functions. However, this may result in overly conservative policies and could not capture complex trajectory-level requirements. Recently, researchers have explored specifying RL objectives using formal languages, which explicitly and unambiguously express trajectory-based task requirements, including safety and liveness properties. Among these, linear temporal logic (LTL) has gained particular popularity [ 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 ] due the automaton-based memory it offers, which ensures history-independence and makes it especially suitable for long-horizon tasks unlike other languages such as signal temporal logic (STL). Specifying desired properties in LTL inherently prevents mismatches between the intended behavior and the behavior learned through reward maximization–one of the most well-known safety challenges in AI [ 51 ] . Although these methods are proven to define the correct RL objectives, the sparse logical rewards make learning extremely difficult, as obtaining a nonzero reward often requires significant exploration. Denser LTL-based rewards provided through heuristics might accelerate learning [ 38 ] ; however, if not carefully designed, they can compromise the correctness of the objective and misguide exploration depending on the environment, ultimately reducing learning efficiency. In this work, we address the challenges of scalable learning with correct objectives for long-horizon learning tasks. We adopt LTL as the specification language, leveraging the intuitive high level language and the automaton-based memory it provides. Unlike prior methods, our approach harnesses gradients from differentiable simulators to facilitate efficient learning directly from LTL specifications while preserving the correctness of the objectives. Our contributions can be summarized as follows: • We propose, to the best of our knowledge, the first approach that accelerates learning from LTL specifications using differentiable simulators. Our approach effectively mitigates the inherent issue of the sparse rewards without sacrificing the expressiveness and correctness that LTL provides. • We introduce soft labeling techniques for continuous environments that yield probabilistic \varepsilon -actions and transitions within the automata derived from LTL, which ensures the differentiability of rewards and states with respect to actions. • Through a serious of experiments, we demonstrate that our approach enables successful learning from LTL specifications in robotic systems, whereas traditional non-differentiable approaches fail to achieve feasible learning. 2 Related Work Safe RL. One common perspective in Safe RL defines safety as the guarantee on the cumulative costs over time within a specified safety budget, which is often modeled using constrained MDPs and has been widely studied [ 8 , 9 , 10 , 11 , 12 ] , relying on additive cost functions and budgets, which may not adequately capture safety in many scenarios. In practice, it is often difficult to assign unambiguous scalar costs reflecting trade-offs between different harmful situations [ 52 ] . Another approach defines safety in terms of avoiding unsafe states and focuses on preventing or modifying unsafe actions via shielding or barrier functions [ 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ] , which only require identification of unsafe states and actions and often easier than designing cost functions [ 53 ] ; however, they can lead to overly conservative control policies [ 54 ] . Moreover, the requirements are often placed over trajectories, which could be more complex than simply avoiding certain states [ 55 ] . Our approach avoids these issues by employing LTL as the specifications language to obtain correct-by-construction RL objectives. RL with Temporal Logics. There is growing increasing in using formal specification languages to encode trajectory-dependent task objectives, especially those involving safety. LTL is widely used due to its expressiveness and well-defined semantics over infinite traces. There has been increasing interest in using formal specification languages to encode task objectives that are trajectory-dependent, particularly those involving safety requirements. LTL has emerged as a widely adopted formalism due to its expressiveness and well-defined semantics over infinite traces. Recent efforts [ 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 ,

Read the complete text