Accelerated Learning with Linear Temporal Logic using Differentiable Simulation Alper Kamil Bozkurt, Calin Belta, Ming C. LinDepartment of Computer ScienceUniversity of Maryland, College Park {bozkurt,calin,lin}@umd.edu Abstract To ensure learned controllers comply with safety and reliability requirements for reinforcement learning in real-world settings remains challenging. Traditional safety assurance approaches, such as state avoidance and constrained Markov decision processes, often inadequately capture trajectory requirements or may result in overly conservative behaviors. To address these limitations, recent studies advocate the use of formal specification languages such as linear temporal logic (LTL), enabling the derivation of correct-by-construction learning objectives from the specified requirements. However, the sparse rewards associated with LTL specifications make learning extremely difficult, whereas dense heuristic-based rewards risk compromising correctness. In this work, we propose the first method, to our knowledge, that integrates LTL with differentiable simulators, facilitating efficient gradient-based learning directly from LTL specifications by coupling with differentiable paradigms. Our approach introduces soft labeling to achieve differentiable rewards and states, effectively mitigating the sparse-reward issue intrinsic to LTL without compromising objective correctness. We validate the efficacy of our method through experiments, demonstrating significant improvements in both reward attainment and training time compared to the discrete methods. † † footnotetext: ∗ equal supervising 1 Introduction The growing demand for artificial intelligence (AI) systems to operate in a wide range of environments underscores the need for systems that can learn through interaction with their environments, without relying on human intervention. Reinforcement learning (RL) has emerged as a powerful tool for training controllers to perform effectively in uncertain settings with intricate, high-dimensional, and nonlinear dynamics. Recent advances in RL have enabled attainment of high-performance controllers in a variety of applications [ 1 ] , such as robotic arm control [ 2 ] , hand manipulation [ 3 ] , legged locomotion [ 4 , 5 ] , navigation in crowded spaces [ 6 ] , and robot-assisted surgery [ 7 ] . Despite the promising results in controlled environments, deploying learned controllers in real-world systems –where malfunctioning can be costly or hazardous– requires not only high performance but also strict compliance with formally specified safety and reliability requirements. Therefore, ensuring that learned controllers meet these critical specifications is essential to fully realize the potential of AI systems in real-world applications. Safety in learning is often modeled with constrained Markov decision processes (MDPs) [ 8 , 9 , 10 , 11 , 12 ] , where the accumulated cost must be within a budget. However, additive cost functions may not reflect real-world safety, as assigning meaningful costs to harms is challenging. Alternative approaches define safety by avoiding unsafe states or actions [ 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ] , which is simpler than designing cost functions. However, this may result in overly conservative policies and could not capture complex trajectory-level requirements. Recently, researchers have explored specifying RL objectives using formal languages, which explicitly and unambiguously express trajectory-based task requirements, including safety and liveness properties. Among these, linear temporal logic (LTL) has gained particular popularity [ 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 ] due the automaton-based memory it offers, which ensures history-independence and makes it especially suitable for long-horizon tasks unlike other languages such as signal temporal logic (STL). Specifying desired properties in LTL inherently prevents mismatches between the intended behavior and the behavior learned through reward maximization–one of the most well-known safety challenges in AI [ 51 ] . Although these methods are proven to define the correct RL objectives, the sparse logical rewards make learning extremely difficult, as obtaining a nonzero reward often requires significant exploration. Denser LTL-based rewards provided through heuristics might accelerate learning [ 38 ] ; however, if not carefully designed, they can compromise the correctness of the objective and misguide exploration depending on the environment, ultimately reducing learning efficiency. In this work, we address the challenges of scalable learning with correct objectives for long-horizon learning tasks. We adopt LTL as the specification language, leveraging the intuitive high level language and the automaton-based memory it provides. Unlike prior methods, our approach harnesses gradients from differentiable simulators to facilitate efficient learning directly from LTL specifications while preserving the correctness of the objectives. Our contributions can be summarized as follows: • We propose, to the best of our knowledge, the first approach that accelerates learning from LTL specifications using differentiable simulators. Our approach effectively mitigates the inherent issue of the sparse rewards without sacrificing the expressiveness and correctness that LTL provides. • We introduce soft labeling techniques for continuous environments that yield probabilistic \varepsilon -actions and transitions within the automata derived from LTL, which ensures the differentiability of rewards and states with respect to actions. • Through a serious of experiments, we demonstrate that our approach enables successful learning from LTL specifications in robotic systems, whereas traditional non-differentiable approaches fail to achieve feasible learning. 2 Related Work Safe RL. One common perspective in Safe RL defines safety as the guarantee on the cumulative costs over time within a specified safety budget, which is often modeled using constrained MDPs and has been widely studied [ 8 , 9 , 10 , 11 , 12 ] , relying on additive cost functions and budgets, which may not adequately capture safety in many scenarios. In practice, it is often difficult to assign unambiguous scalar costs reflecting trade-offs between different harmful situations [ 52 ] . Another approach defines safety in terms of avoiding unsafe states and focuses on preventing or modifying unsafe actions via shielding or barrier functions [ 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ] , which only require identification of unsafe states and actions and often easier than designing cost functions [ 53 ] ; however, they can lead to overly conservative control policies [ 54 ] . Moreover, the requirements are often placed over trajectories, which could be more complex than simply avoiding certain states [ 55 ] . Our approach avoids these issues by employing LTL as the specifications language to obtain correct-by-construction RL objectives. RL with Temporal Logics. There is growing increasing in using formal specification languages to encode trajectory-dependent task objectives, especially those involving safety. LTL is widely used due to its expressiveness and well-defined semantics over infinite traces. There has been increasing interest in using formal specification languages to encode task objectives that are trajectory-dependent, particularly those involving safety requirements. LTL has emerged as a widely adopted formalism due to its expressiveness and well-defined semantics over infinite traces. Recent efforts [ 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 ] derives rewards from LTL specifications for RL, typically by translating LTL into limit-deterministic Büchi automata (LDBAs) and assigning rewards based on acceptance conditions. The memory structure provided by LDBAs supports long-horizon tasks better than alternatives such as STL, which require history-dependent rewards [ 56 ] . However, LTL rewards are often sparse and hinder learning. While heuristic-based dense rewards [ 38 ] attempt to address this, they risk misguiding exploration and compromising correctness. Our approach avoids these pitfalls by leveraging gradients from differentiable simulators to accelerate learning without sacrificing correctness. RL with Differentiable Simulators. Differentiable simulators enable gradient-based policy optimization in RL by computing gradients of states and rewards with respect to actions, using analytic methods [ 57 , 58 , 59 , 60 , 61 ] or auto-differentiation [ 62 , 63 ] . While Backpropagation Through Time (BPTT) is commonly used [ 64 , 65 , 66 , 67 , 68 , 69 ] , it suffers from vanishing or exploding gradients in long-horizon tasks as it ignores the Markov property of states [ 70 ] . To address this, several differentiable RL algorithms have been proposed [ 71 , 72 ] . Short Horizon Actor-Critic (SHAC) [ 73 ] divides long trajectories into shorter segments where BPTT is tractable and bootstraps the remaining trajectory using the value function. Adaptive Horizon Actor-Critic (AHAC) [ 74 ] extends SHAC by dynamically adjusting the segment lengths based on contact information from the simulator. Gradient-Informed PPO [ 75 ] incorporates gradient information into the PPO framework in an adaptive manner. Our approach builds a differentiable, Markovian transition function for LTL-derived automata, making it compatible with all differentiable RL methods. Unlike prior STL-based efforts [ 76 , 77 ] , which rely on non-Markovian rewards and BPTT, our method supports efficient long-horizon learning with full differentiability. 3 Preliminaries and Problem Formulation MDPs. We formalize the interaction between controllers with the environments as MDPs, which can be used for a wide range of robotic systems, including arm manipulation and legged locomotion. Definition 1 . A (differentiable) MDP is a tuple M=(S,A,f,p_{0}) such that S is a set of continuous states; A is a set of continuous actions; f:S\times A\mapsto S is a differentiable transition function; p_{0} is an initial state distribution where p_{0}(s) denotes the probability density for the state s . For a given robotic task, the state space S can be defined by the positions \mathtt{x} and velocities \dot{\mathtt{x}} of relevant objects, body parts, and joints. The action space A may consist of torques applied to the joints. The transition function f captures the underlying system dynamics and outputs the next state via computing the accelerations \ddot{\mathtt{x}} by solving \mathtt{M}\ddot{\mathtt{x}}=\mathtt{J}^{\mathtt{T}}\mathtt{F}(\mathtt{x},\dot{\mathtt{x}})+\mathtt{C}(\mathtt{x},\dot{\mathtt{x}})+\mathtt{T}(\mathtt{x},\dot{\mathtt{x}},a), for a given state s=\langle\mathtt{x},\dot{\mathtt{x}}\rangle\in S and action a\in A . Here, \mathtt{F} , \mathtt{C} , and \mathtt{T} are, respectively, force, Coriolis, and torque functions that can be approximated using differentiable physics simulators. RL Objective. In RL, a given policy \pi:S\mapsto A is evaluated based on the expected cumulative reward (known as return) associated with the paths \sigma\coloneqq s_{0}s_{1}\dots (sequence of visited states) generated by the Markov chain (MC) M_{\pi} induced by the policy \pi . Specifically, for given a reward function R:S\mapsto\mathbb{R} , a discount factor \gamma\in(0,1) , and a horizon H , the return of a path \sigma from time t\in\mathbb{N} is defined as G_{t{:}H}(\sigma)=\sum_{i=t}^{H}\gamma^{i}R(\sigma[i]) . For simplicity, we denote the infinite-horizon return starting from t=0 as G_{H}(\sigma)\coloneqq G_{0{:}H}(\sigma) , and further drop the subscript to write G(\sigma)\coloneqq\lim_{H\to\infty}G_{H}(\sigma) . The discount factor \gamma reduces the value of future rewards to prioritize immediate ones: a reward received after t steps, R(\sigma[t]) , contributes \gamma^{t}R(\sigma[t]) to the return. The objective in RL is to learn a policy that maximizes the expected return over trajectories. Labels. In robotic environments, we define the set of atomic propositions (APs), denoted by \mathtt{A} , as properties of interest that place bounds on functions of the state space. Formally, each AP takes the form \mathtt{a}{\coloneqq}\text{`}g(s){>}0\text{'} , where g:S\mapsto\mathbb{R} is assumed to be a differentiable function mapping a given state to a signal. For example, the function g(\langle\mathtt{x},\dot{\mathtt{x}}\rangle)\coloneqq\dot{\mathtt{x}}^{2}_{\text{max}}-\dot{\mathtt{x}}_{i}^{2} can be used to define an AP that specifies that the velocity of the i -th robotic component must be below an upper bound \dot{\mathtt{x}}_{\text{max}} . The labeling function L:S\mapsto 2^{\mathtt{A}} returns the set of APs that hold true for a given state. Specifically, an AP \mathtt{a}\coloneqq\text{`}g(s)>0\text{'} is included in the label set L(s) of state s – i.e., s is labeled by \mathtt{a} if and only if (iff) g(s)>0 . We also write, with a slight abuse of notation, L(\sigma)\coloneqq L(\sigma[0])L(\sigma[1])\dots to denote the trace (sequences of labels) of a path \sigma . Finally, we write M^{+}{=}(M,L) to denote a labeled MDP. LTL. LTL provides a high-level formal language for specifying the desired temporal behaviors of robotic systems. Alongside the standard operators in propositional logic – negation ( \neg ) and conjunction ( \wedge ) – LTL offers two temporal operators, namely next ( \bigcirc ) and until ( U ). The formal syntax of LTL is defined by the following grammar ( [ 78 ] ): \varphi\coloneqq\mathrm{true}\mid\mathtt{a}\mid\neg\varphi\mid\varphi_{1}\wedge\varphi_{2}\mid\bigcirc\varphi\mid\varphi_{1}\textsf{U}\varphi_{2},\ {\mathtt{a}\in\mathtt{A}}. The semantics of LTL formulas are defined over paths. Specifically, a path \sigma either satisfies \varphi , denoted by \sigma\models\varphi , or not ( \sigma\not\models\varphi ). The satisfaction relation is defined recursively as follows: \sigma\models\varphi ; if \varphi=\mathtt{a} and \mathtt{a}\in L(\sigma[0]) (i.e., \mathtt{a} immediately holds); if \varphi=\neg\varphi^{\prime} and \sigma\not\models\varphi^{\prime} ; if \varphi=\varphi_{1}\wedge\varphi_{2} and (\sigma\models\varphi_{1})\wedge(\sigma\models\varphi_{2}) ; if \varphi=\varphi_{1}\textsf{U}\varphi_{2} and there exists t\geq 0 such that \sigma[t{:}]\models\varphi_{2} and for all 0\leq i0\text{'} , belongs to the label L(s) of a state s . Formally, we define this probability as: \displaystyle\text{Pr}(\mathtt{a}\in L(s))=\text{Pr}(g(s)>0)\coloneqq h(g(s))=\frac{1}{1+\text{exp}(-g(s))}. (3) Although we use the widely adopted sigmoid function here † † For the correctness of LTL, \text{Pr}(g(s)>0) must be exactly 0 or 1 for values below or above certain thresholds. In practice, this is not an issue, as overflow behavior of sigmoid ensures this condition is satisfied , any differentiable cumulative distribution function (CDF) h:\mathbb{R}\mapsto[0,1] could be applied. Building upon these probabilities, we define the probability associated with a label l as follows: \displaystyle\text{Pr}(L(s)=l)=\prod_{\mathtt{a}\in l}\text{Pr}(\mathtt{a}\in L(s))\prod_{\mathtt{a}\not\in l}(1-\text{Pr}(\mathtt{a}\in L(s))). (4) These probabilistic labels induce probabilistic automaton transitions, causing the controller to observe automaton states probabilistically. Consequently, instead of modeling automaton states as deterministic indicator vectors in product MDPs, we represent them as probabilistic superpositions over all possible automaton states. By doing so, we design differentiable transitions and rewards within the product MDP. Let f_{L}:S{\times}\mathbf{Q}\mapsto\mathbf{Q} denote the function that updates the automaton state probabilities based on the LDBA transitions triggered by probabilistic labels, and let \mathbf{q} denote the vector where each element \mathbf{q}_{q} is the probability of being in automaton state q , then we can formally define: \displaystyle f_{L}(\langle s,\mathbf{q}\rangle)=\mathbf{q}^{\prime}\ \ \text{\scriptsize where}\ \ \mathbf{q}^{\prime}_{q^{\prime}}{=}\hskip-2.0pt\sum_{q}\hskip-1.0pt\mathbf{q}_{q}\hskip-4.0pt\sum_{l\in L_{q,q^{\prime}}}\hskip-4.0pt\text{Pr}(L(s){=}l)\ \ \text{\scriptsize and }\ L_{q,q^{\prime}}{\coloneqq}\{l\mid q^{\prime}{=}\delta(q,l)\}. (5) Intuitively, the probability of transitioning to a subsequent automaton state q^{\prime} is computed by summing probabilities across all current automaton states q and labels l\in L_{q,q^{\prime}} capable of leading to state q^{\prime} . This computation can be efficiently done through differentiable matrix multiplication. The remaining hurdle is the binary \varepsilon -actions available to the controller, which trigger \varepsilon -transitions in the LDBA. Similarly to the soft labels approach, \varepsilon -actions can become differentiable by representing the probabilities of the \varepsilon -transitions to be triggered. Let f_{\varepsilon}:\mathbf{Q}\times\mathbf{Q}\mapsto\mathbf{Q} denote the function updating automaton state probabilities based on the \varepsilon -action taken, and let \mathbf{q}^{\varepsilon} denote the vector whose elements indicate the probabilities of taking the \varepsilon -actions leading to the corresponding automaton states, we then define: \displaystyle f_{\varepsilon}(\mathbf{q},\mathbf{q}^{\varepsilon})\hskip-1.0pt=\hskip-1.0pt\mathbf{q}^{\prime}\text{\scriptsize\ where \ }\mathbf{q}^{\prime}_{q^{\prime}}{=}\hskip-8.0pt\sum_{q\in Q_{\varepsilon\hskip-1.0pt,q^{\prime}}}\hskip-9.0pt\mathbf{q}_{q}\mathbf{q}^{\varepsilon}_{q^{\prime}}{+}\hskip-8.0pt\sum_{q\in\overline{Q}_{q^{\prime}\hskip-2.0pt,\varepsilon}}\hskip-9.0pt\mathbf{q}_{q^{\prime}}\mathbf{q}^{\varepsilon}_{q},\ Q_{\varepsilon\hskip-1.0pt,q^{\prime}}\hskip-1.0pt{\coloneqq}\hskip-1.0pt\{q\hskip-1.0pt\mid\hskip-1.0ptq^{\prime}\hskip-1.0pt{\in}\delta(q\hskip-0.5pt,\hskip-0.5pt\varepsilon)\}\hskip-1.0pt,\ \overline{Q}_{q^{\prime}\hskip-2.0pt,\varepsilon}\hskip-1.0pt{\coloneqq}\hskip-1.0pt\{q\hskip-1.0pt\mid\hskip-1.0ptq{\not\in}\delta(q^{\prime}\hskip-2.5pt,\hskip-0.5pt\varepsilon)\}. (6) Conceptually, the probability of transitioning to automaton state q^{\prime} involves two scenarios: (the first summation in ( 6 )) the probability of moving to q^{\prime} via valid \varepsilon -transitions, and (the second summation in ( 6 )) the probability of remaining in q^{\prime} after trying to leave from q^{\prime} via nonexistent \varepsilon -transitions. These vector computations can be efficiently performed in a differentiable manner. We can formulate the complete transition function \mathfrak{f} by composing f_{L} , f_{\varepsilon} , and f as follows: \displaystyle\mathfrak{f}(\langle s,\mathbf{q}\rangle,\langle a,\mathbf{q}^{\varepsilon}\rangle)\coloneqq\langle f(s,a),f_{L}(\big\langle s,f_{\varepsilon}(\mathbf{q},\mathbf{q}^{\varepsilon})\rangle)\big\rangle. (7) This transition function first executes the \varepsilon -actions, then performs the LDBA transitions triggered by state labels to update the automaton state probabilities, while applying the given action to update the MDP states. The function \mathfrak{f} is fully differentiable with respect to s , \mathbf{q} , a , and \mathbf{q}^{\varepsilon} . We can now obtain a reward \mathfrak{R}:\mathbf{Q}\mapsto(0,1) and a discounting function \mathfrak{D}:\mathbf{Q}\mapsto(0,1) that are also differentiable with respect state and actions as follows: \displaystyle\mathfrak{R}(\langle s,\mathbf{q}\rangle)\coloneqq(1-\beta)\sum_{q\in B}\mathbf{q}_{q},\quad\mathfrak{D}(\mathbf{q})\coloneqq\beta\sum_{q\in B}\mathbf{q}_{q}+\gamma\sum_{q\not\in B}\mathbf{q}_{q} (8) These differentiable reward, discounting and functions allow us to obtain first-order gradient estimates \nabla_{\theta}^{1}J(\theta)\coloneqq\mathbb{E}_{\sigma\sim M_{\pi_{\theta}}}\left[\nabla_{\theta}G_{H}(\sigma)\right] which are known to exhibit lower variance compared to zeroth-order estimates [ 73 ] . Such first-order estimates can be effectively utilized by differentiable RL algorithms to accelerate learning. In the following example, we illustrate employing these lower-variance gradient estimates is particularly crucial when learning from LTL rewards. Figure 1 : LTL Returns and Derivatives. Left : The parking scenario where the car must brake to stop in the parking area without entering the grass field ( \varphi_{p} ). Middle : LTL satisfaction probability and return estimates from discrete and differentiable LTL formulations as functions of deceleration. Right : LTL return gradients with respect to deceleration and their standard deviation. The key challenge in learning from LTL arises from slightly-sloped regions and sharp changes in the returns produced by discrete LTL rewards. Our differentiable LTL approach not only smooths these abrupt changes but also enables the use of low-variance first-order gradient estimates essential for effective learning in slightly-sloped regions . Parking Example. Consider a parking scenario in which the vehicle starts with an initial velocity of v_{0}=10 m/s. The controller applies the brakes with a constant deceleration a\in[0\text{ m/s}^{2},10\text{ m/s}^{2}] over the next 10 seconds, with the goal of bringing the car to rest inside the parking area. For safety, the vehicle must not enter the grass field before reaching the parking zone on the right-hand side. We formalize these requirements in LTL as \varphi_{p}{=}\lozenge\square\texttt{park}\wedge\square\neg\texttt{grass} where the parking area and the grass field are defined as \texttt{park}\coloneqq(x{>}10\text{ m}\wedge x{<}20\text{ m})\vee(x{>}30\text{ m}\wedge x{<}40\text{ m}) and \texttt{grass}\coloneqq x{>}20\text{ m}\wedge x{<}30\text{ m} , respectively. Figure 1 illustrates this task, including satisfaction probabilities, returns, and gradients with respect to deceleration. The satisfaction probability is 1 for deceleration values between 2.5\text{ m/s}^{2} and 5.0\text{ m/s}^{2} , and 0 outside this range. The differentiable LTL returns closely match the discrete ones, except near the boundaries of the satisfaction region, where the differentiable version produces smoother transitions. This smoothness is particularly evident in the gradient plots. Although differentiable LTL rewards yield smoother return curves, learning remains challenging due to the small gradient magnitudes across most of the parameter space except near the satisfaction boundaries. For instance, in the region between 0.0\text{ m/s}^{2} and 2.5\text{ m/s}^{2} , the returns increase with deceleration, but noisy gradient estimates can still lead the learner away from the satisfaction region. Therefore, obtaining low-variance gradient estimates is especially beneficial when learning from LTL, where most of the landscape requires sharper gradients for effective optimization. 5 Experiments In this section, through simulated experiments, we show learning from the differentiable LTL rewards provided by our approach is significantly faster than learning from standard discrete LTL rewards. Algorithm 1 Differentiable RL with LTL 0: MDP M , LTL formula \varphi , Policy \pi_{\theta} Derive LDBA A_{\varphi} and APs \mathtt{A} from \varphi Derive \mathfrak{f} ( 7 ) and \mathfrak{R},\mathfrak{D} ( 8 ) from A_{\varphi} while True do Initialize \mathbf{q}^{(0)}{\sim}A_{\varphi} , s^{(0)}{\sim}M , G{\leftarrow}0 for t=1,2,...,H do Get action \langle a,\mathbf{q}^{\varepsilon}\rangle\sim\pi_{\theta}(\langle s^{(t\text{-}1)},\mathbf{q}^{(t\text{-}1)}\rangle) Execute \varepsilon -action \mathbf{q}^{\prime}{\leftarrow}f_{\varepsilon}(\mathbf{q},\mathbf{q}^{\varepsilon}) Execute label transition \mathbf{q}^{(t)}{\leftarrow}f_{L}(\langle s,\mathbf{q^{\prime}}\rangle) Execute MDP action s^{(t)}\leftarrow f(s,a) Compute reward r\leftarrow\mathfrak{R}(\mathbf{q^{(t)}}) Update return G\leftarrow G+\mathfrak{D}(\mathbf{\mathbf{q}})\cdot r end for Train \pi_{\theta} using differentiable return G end while Implementation Details. We implemented our approach in Python utilizing the PyTorch-based differentiable physics simulator dFlex introduced in [ 73 ] . We used an NVIDIA GeForce RTX 2080 GPU, 4 Intel(R) Xeon(R) Gold 5218 CPU cores, and 32 gigabytes memory for each experiment. Specifically, we generate the automaton description using Owl [ 81 ] and parse it using Spot [ 82 ] . We then construct reward and transition tensors from the automata. We then compute the probabilities for each observations as explained in the previous section using a sequence of differentiable vector operations using PyTorch. Lastly, using the constructed transition and reward tensors, we update the automaton states and provide rewards. The overall approach is summarized in Algorithm 1 . Baselines. We use two widely adopted and representative state-of-the-art (SOTA) model-free RL algorithms as our baseline non-differentiable RL methods ( \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs): the on-policy Proximal Policy Optimization (PPO) [ 83 ] and the off-policy Soft Actor-Critic (SAC) [ 84 ] . For differentiable RL baselines ( \partial RLs), we employ SHAC and AHAC, which, to the best of our knowledge, represent the SOTA in this category. For each environment and baseline, we adopted the tuned hyperparameters in [ 74 ] . Metric. We evaluate performance in terms of the collected LTL rewards averaged over 5 seeds since they can serve as proxies for satisfaction probabilities. We considered two criteria: (1) the maximum return achieved and (2) the speed of convergence. To maintain consistency, we used differentiable LTL rewards across all baselines as, for non-differentiable baselines, we observed no performance difference between the differentiable and discrete LTL rewards. Figure 2 : Task Specification with LTL. This figure illustrates a Cheetah policy learned by SHAC using differentiable rewards derived via our approach from the LTL formula \varphi_{\text{legged}} ( 10 ), which specifies accelerating forward, stopping, and maintaining a safe tip-to-ground distance. Specifying the desired behaviors of robots using the high-level language LTL provides is an intuitive alternative to manually designing reward functions, which often require extensive domain expertise and risk unintended behaviors. Enabling learning directly from LTL unlocks new possibilities for robust, safe, and flexible robotic applications. CartPole. The CartPole environment consists of a cart that moves along a one-dimensional track, with a pole hinged to its top that can be freely rotated by applying torque. The system yields a 5-dimensional observation space and a 1-dimensional action space. The control objective is to move the tip of the pole through a sequence of target positions while maintaining the cart within a desired region as much as possible and ensuring the velocity of the cart always remains within safe boundaries. We capture these requirements in LTL as follows: \displaystyle\hskip-3.01125pt\varphi_{\text{cartpole}}{=}\underbrace{\square\text{`$|${cart\_vx}$|{<}v_{0}$\hskip-2.0pt'}}_{\text{safety}}{\wedge}\underbrace{\square\lozenge\text{`$|${cart\_x}$|{<}x_{0}$\hskip-2.0pt'}}_{\text{repetition}}{\wedge}\underbrace{\lozenge\big(\text{`$|${pole\_z}-$z_{0}|{<}\Delta$\hskip-2.0pt'}{\wedge}\lozenge\text{`$|${pole\_z}-$z_{1}|{<}\Delta$\hskip-2.0pt'}\big)}_{\text{reachability \& sequencing}}. (9) Here, cart_x , cart_vx , and pole_ z represent the cart position, the cart velocity, and the pole height respectively. This formula demonstrates how LTL can be leveraged to encode both complex safety constraints and performance objectives. Specifically, we set x_{0}=10 m, v_{0}=10 m/s as boundaries, z_{0}=-1 m, z_{1}=1 m as the target positions, and \Delta=25 cm as the allowable deviation. Legged Robots. We consider three legged-robot environments: Hopper, Cheetah, and Ant. The Hopper environment features a one-legged robot with 4 components and 3 joints, resulting in a 10-dimensional state space and a 3-dimensional action space. The Cheetah environment consists of a two-legged robot with 8 components and 6 joints, yielding a 17-dimensional state space and a 6-dimensional action space. The Ant environment includes a four-legged robot with 9 components and 8 joints, producing a 37-dimensional state space and an 8-dimensional action space. In all three environments, the control task requires always keeping the torso/tip of the robot above a critical safety height, maintaining a certain distance between the torso/tip and the critical height as often as possible, and accelerating the robot forward, and then bringing the robot to a full stop. We formalize this task in LTL as follows: \displaystyle\varphi_{\text{legged}}{=}\underbrace{\square\text{`{torso\_z}${>}z_{0}$\hskip-2.0pt'}}_{\text{safety}}{\wedge}\underbrace{\square\lozenge\text{`{torso\_z}${>}z_{1}$\hskip-2.0pt'}}_{\text{repetition}}{\wedge}\underbrace{\lozenge\big(\text{`{torso\_vx}${>}v_{1}$\hskip-2.0pt'}{\wedge}\lozenge\text{`{torso\_vx}${<}v_{0}$\hskip-2.0pt'}\big)}_{\text{reachability \& sequencing}}. (10) Here, torso_z and torso_vx denote the height and horizontal velocity of the robots. This formula captures several key aspects of LTL, including, safety, reachability, sequencing, and repetition. The values of z_{0} and z_{1} were chosen based on the torso height of each robot in their referential system. Specifically, we used z_{0}=-110 cm, z_{1}=-105 cm for Hopper; z_{0}=-75 cm, z_{1}=-70 cm for Cheetah; and z_{0}=0 cm, z_{1}=5 cm for Ant, where z_{0} denotes the critical safety height and z_{1} represents a safe margin above it. We set v_{1}=1 m/s, v_{1}=3 m/s, and v_{1}=1.5 m/s for Hopper, Cheetah, and Ant, respectively, reflecting movement speeds relatively challenging yet achievable for each of the robot. For deceleration, we set v_{0}=0 m/s for all the environments. An illustration of a policy learned from this specification for Cheetah is provided in Figure 2 . Figure 3 : Comparison Across Environments: Differentiable vs. Discrete LTL Rewards. The wider plots show the learning curves of all baseline algorithms, while the narrower plots on the right display the maximum returns achieved after 100 M steps. All results are averaged over 5 random seeds, and the curves are smoothed using max and uniform filters for visual clarity. The reported returns, bounded between 0 and 1, serve as proxies for the probability of satisfying the LTL specifications. In all the environments algorithms utilizing differentiable LTL rewards (SHAC, AHAC) rapidly learn near-optimal policies, whereas those relying on discrete LTL rewards (PPO, SAC), display high variance, converge slowly, or getting stuck with sub-optimal/near-zero-return policies. Results. Figure 3 presents our simulation results. Across all environments, \partial RL algorithms that leverage our differentiable LTL rewards consistently outperform \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RL algorithms in terms of both maximum return achieved and learning speed from the LTL specifications. CartPole. The safety specification induces an automaton with three states, each having 64 transitions–but only one of these transitions yields a reward. This extreme sparsity, even in a low-dimensional state space, severely hinders the learning process for \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs, as shown in the leftmost plot of Figure 3 . In contrast, \partial RL algorithms leverage the gradients provided by differentiable rewards, enabling them to efficiently learn policies that nearly satisfy the LTL specification. Specifically, \partial RLs converge to near-optimal policies ( \text{Pr}{>}0.8 ) within just 20 M steps, whereas \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs (SAC: all seeds; PPO: one seed) fail to learn any policy that achieves meaningful reward, even after 100 million (M) steps. Legged Robots. As we move to environments with higher-dimensional state spaces–10, 17, and 37 dimensions for Hopper, Cheetah, and Ant, respectively–even relatively simple LTL specifications pose a significant challenge for \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs. The automata derived from the LTL specifications in these environments consists of four states, each with 16 transitions, of which four transitions in the third state yield rewards. Reaching this state, however, requires extensive blind exploration of the state space, making it significantly hard for \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs to learn optimal control policies. On the other hand, \partial RLs, guided by LTL reward gradients, quickly identify high-reward regions of the state space and learn effective policies. For Hopper, \partial RLs converge to near-optimal policies ( \text{Pr}{>}0.8 ) within 20 M steps, while PPO requires the full 100 M steps to converge, and one SAC seed gets trapped in a local optimum. For Cheetah, \partial RLs attain optimal performance ( \text{Pr}{>}0.9 ), whereas PPO converges to a suboptimal policy even after 100 M steps, and SAC consistently fails by getting stuck in poor local optima. For Ant, \partial RLs again learn near-optimal policies rapidly, while \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs converge only to suboptimal policies. Figure 4 : Ablation Study for LTL. The maximum returns obtained after 100 M steps for simplified LTL formulas ( 12 ), averaged over 5 seeds. Returns (0 to 1) indicate LTL satisfaction probabilities. Under these simpler specifications, both \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.2389pt\kern-4.48618pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.2389pt\kern-4.48618pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.09709pt\kern-4.0584pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.0618pt\kern-2.73613pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs and \partial RLs successfully learn near-optimal policies. However, as shown in Figure 3 , the performance of discrete \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.2389pt\kern-4.48618pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.2389pt\kern-4.48618pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.09709pt\kern-4.0584pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.0618pt\kern-2.73613pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs degrades dramatically with increasing LTL complexity—unlike differentiable \partial RLs, which maintain reasonable performance by leveraging the LTL rewards differentiability. Ablation Study. To isolate the impact of differentiability of LTL rewards from inherent environment properties, we conduct an ablation study comparing \partial RLs and \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs under simplified LTL specifications. Specifically, we use reduced versions of the LTL formulas from our earlier experiments: \displaystyle\varphi^{\prime}_{\text{cartpole}} \displaystyle\coloneqq\lozenge\text{`$|${pole\_z}-$z_{0}|{<}\Delta$'} (11) \displaystyle\varphi^{\prime}_{\text{legged}} \displaystyle\coloneqq\lozenge\text{`{torso\_vx}${>}v_{1}$\hskip-2.0pt'} (12) using z_{0}=-1 m, \Delta=25 cm for Cartpole, and v_{1}=50 cm/s for all the legged-robot environments. These simplified formulas yield one-state automata with 4 and 2 transitions, respectively, of which one is accepting. As such, they lack the complexity that makes learning from LTL challenging. Figure 4 presents the maximum returns obtained for these simplified specifications. Each of the baselines, regardless of differentiability, learns an optimal policy ( \text{Pr}{>}0.9 ) for all the environments. However, when comparing these results to those in Figure 3 , we observe only a minor performance drop for \partial RLs, whereas the performance of \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs degrades dramatically—for some cases, from near satisfaction to complete failure—as LTL complexity increases. These results support our hypothesis that the performance advantage of \partial RLs over \mathchoice{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\displaystyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.65451pt\kern-5.27776pt$\textstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 2.13524pt\kern-4.45831pt$\scriptstyle\not$\hss}{\partial}}}{\mathrel{\hbox to0.0pt{\kern 1.8724pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\partial}}} RLs arises primarily from leveraging the differentiability of LTL rewards provided by our approach, rather than from environment-specific properties. 6 Discussion and Limitations Our approach accelerates learning from LTL specifications by leveraging differentiable RL algorithms that utilize gradients provided by differentiable simulators. Therefore, the overall performance of our method is inherently influenced by the quality and efficiency of the underlying simulators and RL algorithms. For example, if a simulator provides poor gradient information or computes gradients slowly, the learning process will be significantly slowed down. Another issue is the reliance on hyperparameters. Although we adopt tuned hyperparameters from existing work, applying our approach to new environments may require additional hyperparameter tuning. A further challenge lies in the formalization of LTL specifications. While LTL offers a more intuitive and structured way to specify tasks compared to manual reward engineering, it still requires familiarity with formal logic and sufficient domain knowledge to define meaningful bounds. Finally, our method introduces an additional hyperparameter: the CDF used for probability estimation, which must also be tuned for optimal performance. 7 Conclusion In this work, we tackle the critical challenge of scalable RL for robotic systems under long-horizon, formally specified tasks. By adopting LTL as our specification framework, we ensure objective correctness and avoid the reward misspecification issues commonly encountered in conventional RL approaches. To overcome the learning inefficiencies caused by sparse logical rewards, we propose a novel method that leverages differentiable simulators, enabling gradient-based learning directly from LTL objectives without compromising their expressiveness or correctness. Our approach introduces soft labeling techniques that preserve the differentiability through the transitions of automata derived from LTL formulas, resulting in end-to-end differentiable learning framework. Through a series of simulated experiments, we demonstrate that our method substantially accelerates learning compared to SOTA non-differentiable baselines, paving the way for more reliable and scalable deployment of autonomous robotic systems in complex real-world environments. Appendix A Paths An MDP begins in an initial state sampled from the initial state distribution s_{0}\sim p_{0}(\cdot) and evolves by transitioning from a current state s to a next state s^{\prime} through an action a , as determined by the transition function: s^{\prime}=f(s,a) . In each state s , the policy observes a set of atomic propositions provided by the labeling function L(s) . The sequence of visited states is called a path and is formally defined below: Definition 3 . A path of an MDP M is defined as an infinite sequence of states \sigma=s_{0}s_{1}\dots , where each s_{i}\in S , such that p_{0}(s_{0})>0 and for every t>0 , there exists an action a_{t}\in A with f(s_{t},a_{t})=s_{t+1} . We denote the t -th state in the sequence as \sigma[t] , the prefix up to t as \sigma[{:}t]=s_{0}s_{1}\dots s_{i} , and the suffix starting from t+1 as \sigma[t{+}1{:}]=s_{t+1}s_{t+2}\dots . The corresponding sequence of labels for \sigma is referred to as the trace, defined by L(\sigma)\coloneqq L(s_{0})L(s_{1})\dots . See [ 85 ] for more details. Appendix B \omega -Automata An LTL formula \varphi can be translated into a finite-state automaton that operates over infinite paths, known as an \omega -automaton. We denote the automaton corresponding to a specific formula \varphi by \mathcal{A}\varphi . An \omega -automaton \mathcal{A}\varphi accepts a path \sigma if and only if \sigma\models\varphi , and rejects it otherwise. We begin by formally introducing a general type of \omega -automaton, called a nondeterministic Rabin automaton (NRA), which can be systematically derived from any LTL formula [ 78 ] . We then focus on specific subclasses of NRAs that are particularly relevant to our work. Definition 4 . A nondeterministic Rabin automaton (NRA) derived from an LTL formula \varphi is defined as a tuple \mathcal{A}\varphi=(Q,q_{0},\Sigma,\delta,\textnormal{Acc}) , where: • Q is a finite set of automaton states; • q_{0}\in Q is the initial automaton state; • \Sigma=2^{\mathtt{A}} is the input alphabet, where \mathtt{A} is the set of atomic propositions; • \delta:Q\times(\Sigma\cup\{\varepsilon\})\rightarrow 2^{Q} is the transition function, which is complete and deterministic on \Sigma (i.e., |\delta(q,w)|=1 for any q\in Q and w\in\Sigma ), but may include nondeterministic \varepsilon -transitions (i.e., it is possible that |\delta(q,\varepsilon)|=0 or |\delta(q,\varepsilon)|>1 ); • Acc is a set of k accepting pairs \{(B_{i},C_{i})\}_{i=1}^{k} where B_{i},C_{i}\subseteq Q for i\in\{1,\dots,k\} . Given an infinite word \omega=w_{0}w_{1}\dots , a run of the automaton is a sequence of transitions \tau_{\omega}=(q_{0},w_{0},q_{1}),(q_{1},w_{1},q_{2}),\dots such that q_{t+1}\in\delta(q_{t},w_{t}) for all t\geq 0 . The word \omega is accepted if there exists such a run and at least one accepting pair (B_{t},C_{t}) satisfying the Rabin condition: the run visits some states in B_{t} infinitely often and all states in C_{t} only finitely often. This acceptance condition is known as the Rabin condition, which can be formalized as \omega\models\varphi\iff\exists t:\textnormal{Inf}(\tau_{\omega}){\cap}B_{i}{\neq}\varnothing\wedge\textnormal{Inf}(\tau_{\omega}){\cap}C_{i}{=}\varnothing where \textnormal{Inf}(\tau_{\omega}) denotes the set of states visited infinitely many times in the run \tau_{\omega} . Similarly, a path \sigma is considered accepted by the automaton if its trace L(\sigma) forms a word accepted by \mathcal{A}_{\varphi} . We focus on a specific subclass of NRAs known as limit-deterministic Büchi automata (LDBAs), which feature a simplified acceptance criterion. Despite their reduced complexity, LDBAs retain the full expressive power of general NRAs and can be systematically derived from LTL formulas [ 86 , 79 ] . Definition 5 . An LDBA is an NRA defined as \mathcal{A}_{\varphi}=(Q,q_{0},\Sigma,\delta,\textnormal{Acc}) that satisfies the following: • The acceptance condition consists of a single pair with an empty second set, i.e., \textnormal{Acc}={(B,\varnothing)} , meaning that the run must visit some states in B infinitely often. This is known as the Büchi condition, and for simplicity, we denote this acceptance condition by the set of accepting automaton states itself, i.e., \textnormal{Acc}=B . • The state space Q is partitioned into two disjoint subsets: an initial component Q_{I} and an accepting component Q_{A} , satisfying: – All accepting states are contained in Q_{A} , i.e., B\subseteq Q_{A} ; – All transitions within Q_{A} are deterministic (no \varepsilon -transitions), i.e., for any q{\in}Q_{A} , \delta(q,\varepsilon){=}\varnothing ; – Transitions cannot go from Q_{A} to Q_{I} , i.e., for all q\in Q_{A} and w\in\Sigma , \delta(q,w)\subseteq Q_{A} . The defining feature of LDBAs is that accepting runs must eventually enter the accepting component Q_{A} and remain there permanently. Once this transition occurs, all subsequent behavior is deterministic–this property is known as limit-determinism. LDBAs can be constructed such that every \varepsilon -transition leads directly into Q_{A} , which ensures at most one such transition occurs along any execution path. This construction, combined with limit-determinism, makes LDBAs particularly well-suited for quantitative model checking in MDPs [ 79 ] unlike general nondeterministic automata. Thus, we assume all LDBAs under consideration possess this structure. More information can be found in [ 85 ] . Appendix C LTL Examples LTL can be used to specify temporal properties of a wide range of robotics tasks: • safety; e.g., “always avoid obstacles and remain below the joint angle and velocity thresholds”: \displaystyle\varphi=\square\Big(\bigwedge_{\text{obstacle}_{i}}\text{dist\_to\_obstacle}_{i}>0\bigwedge_{\text{joint\_angle}_{i}}\text{joint\_angle}_{i}<\beta_{\text{max}}\bigwedge_{\text{joint\_vel}_{i}}\text{joint\_vel}_{i}<\dot{\beta}_{\text{max}}\Big); (13) • reachability; e.g.; “accelerate to a target velocity in x direction”: \displaystyle\varphi=\lozenge\text{torso\_vel}_{x}>v_{\text{target}}; (14) • sequencing; e.g.; “move to the position 1, then move to the position 2, and after that move to the position 3”: \displaystyle\varphi=\lozenge\Big(\text{dist\_to\_pos}_{1}<\delta_{\text{tol}}\wedge\lozenge\big(\text{dist\_to\_pos}_{2}<\delta_{\text{tol}}\wedge\lozenge(\text{dist\_to\_pos}_{3}<\delta_{\text{tol}})\big)\Big); (15) • repetition; e.g.; “repeatedly monitor the region 1 and the region 2”: \displaystyle\varphi=\square\lozenge\text{dist\_to\_region}_{1}<\delta_{\text{tol}}\wedge\square\lozenge\big( \displaystyle\text{dist\_to\_region}_{2}<\delta_{\text{tol}}\big). (16) Appendix D Finite-Memory Policies Definition 6 . A finite-memory policy for an MDP M is defined as a tuple \pi=(\mathfrak{M},\mathfrak{m}_{0},\mathfrak{T},\mathfrak{a}) , where: • \mathfrak{M} is a finite set of modes (memory states); • \mathfrak{m}_{0}\in\mathfrak{M} is the initial mode; • \mathfrak{T}:\mathfrak{M}\times S\times\mathfrak{M}\rightarrow[0,1] is a probabilistic mode transition function such that for any current mode \mathfrak{m} and state s , the probabilities over next modes sum to 1, i.e., \sum_{\mathfrak{m}^{\prime}\in\mathfrak{M}}\mathfrak{T}(\mathfrak{m}^{\prime}\mid\mathfrak{m},s)=1 ; • \mathfrak{a}:\mathfrak{M}\times S\times A\rightarrow[0,1] is a probabilistic action selection function that assigns a probability to each action a given the current mode \mathfrak{m}\in\mathfrak{M} and state s\in S . A finite-memory policy acts as a finite-state machine that updates its internal mode (memory state) as states are observed, and specifies a distribution over actions based on both the current state and mode. The action at each step is thus selected according not only the current state but also the current memory state of the policy. In contrast to standard definitions of finite-memory policies (e.g., [ 80 , 78 ] ), which typically assume deterministic mode transitions, this definition permits probabilistic transitions between modes. For further details, please refer to [ 85 ] . Appendix E Reinforcement Learning Objective The objective of policy gradient algorithms for a given horizon H is to find the optimal policy parameters \theta^{*}\coloneqq\mathrm{argmax}_{\theta}J(\theta) , where \displaystyle J(\theta)\coloneqq\mathbb{E}_{\sigma\sim M_{\pi_{\theta}}}[G_{H}(\sigma)]. (17) If the model is available and differentiable, one can utilize the following first-order gradients for optimization: \displaystyle\nabla_{\theta}^{[1]}J(\theta)\coloneqq\mathbb{E}_{\sigma\sim M_{\pi_{\theta}}}\left[\nabla_{\theta}G_{H}(\sigma)\right], (18) Alternatively, via the policy gradient theorem, one can employ the following model-free zeroth-order policy gradients \displaystyle\nabla_{\theta}^{[0]}J(\theta)\coloneqq\mathbb{E}_{\sigma\sim M_{\pi_{\theta}}}\left[G_{H}(\sigma)\sum_{t=0}^{H-1}\nabla_{\theta}\textnormal{log}\pi_{\theta}(a_{t}|s_{t})\right]. (19) Both gradient estimates can be approximated using Monte Carlo sampling: \displaystyle\bar{\nabla}_{\theta}^{[1]} \displaystyle=\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta}G_{H}(\sigma_{i}), \displaystyle\sigma_{i}\sim M_{\pi_{\theta}}; (20) \displaystyle\bar{\nabla}_{\theta}^{[0]} \displaystyle=\frac{1}{N}\sum_{i=1}^{N}G_{H}(\sigma_{i})\sum_{t=0}^{H-1}\nabla_{\theta}\textnormal{log}\pi_{\theta}(a_{t}|s_{t}), \displaystyle\sigma_{i}\sim M_{\pi_{\theta}}. (21) Appendix F Parking Example: Differentiable RL vs Non-Differentiable RL Figure 5 : Convergence speed comparison of stochastic gradient descent algorithms using \bar{\nabla}_{\theta}^{[0]} and \bar{\nabla}_{\theta}^{[1]} for the parking example ( N=10 ). Appendix G LTL Specifications and Automata used in Experiments • CartPole: G("position_x>-10" & "position_x<10") & G("velocity_x>-10.0" & "velocity_x<10.0") & F("cos_theta<-0.5" & F"cos_theta>0.5") • Hopper: G"torso_height>-11.0" & GF"torso_height>-10.5" & F("torso_velocity_x>1.0" & F"torso_velocity_x<0") • Cheetah: G"tip_height>-7.5" & GF"tip_height>-7.0" & F("tip_velocity_x>3.0" & F"tip_velocity_x<0") • Ant: G"torso_height>0.0" & GF"torso_height>0.5" & F("torso_velocity_x>1.5" & F"torso_velocity_x<0") Figure 6 : The \omega -automaton derived from \varphi_{\text{cartpole}} from ( 9 ). Figure 7 : The \omega -automaton derived from \varphi_{\text{legged}} from ( 10 ) for the Ant environment.