LTLCodeGen: Code Generation of Syntactically Correct Temporal Logic for Robot Task Planning Behrad Rabiei *, Mahesh Kumar A.R. *, Zhirui Dai, Surya L.S.R. Pilla, Qiyue Dong, Nikolay Atanasov Abstract This paper focuses on planning robot navigation tasks from natural language specifications. We develop a modular approach, where a large language model (LLM) translates the natural language instructions into a linear temporal logic (LTL) formula with propositions defined by object classes in a semantic occupancy map. The LTL formula and the semantic occupancy map are provided to a motion planning algorithm to generate a collision-free robot path that satisfies the natural language instructions. Our main contribution is LTLCodeGen, a method to translate natural language to syntactically correct LTL using code generation. We demonstrate the complete task planning method in real-world experiments involving human speech to provide navigation instructions to a mobile robot. We also thoroughly evaluate our approach in simulated and real-world experiments in comparison to end-to-end LLM task planning and state-of-the-art LLM-to-LTL translation methods. I Introduction The ability to understand and execute tasks specified in natural language is an important aspect of enabling autonomous robot assistants in our daily lives. A fundamental challenge is to translate high-level natural-language task descriptions into low-level, executable robot actions. Large Language Models (LLMs), such as GPT-4 [ 1 ] , LLaMA-2 [ 2 ] , and DeepSeek-R1 [ 3 ] , have demonstrated remarkable natural language proficiency and language-based task reasoning [ 4 , 5 ] and planning capabilities [ 6 ] . LLMs play different roles in the planning of robot task executions for natural language specifications [ 7 , 4 , 8 , 9 , 10 , 11 , 12 , 5 , 13 ] . An LLM can be a context extractor, which connects natural language concepts to real-world objects and locations, a process known as symbol grounding [ 14 ] . LLMs are also used as translators, for example, to convert natural language instructions into linear temporal logic (LTL) formulas [ 7 ] , signal temporal logic (STL) [ 5 , 13 ] , planning domain definition language (PDDL) [ 13 ] , or executable code [ 8 , 10 , 15 ] . An LLM can also operate as a task scheduler, which decomposes the task into sub-tasks, infers the order and the dependencies among the sub-tasks, and distributes them to different robots [ 5 , 11 ] . LLMs also may cooperate with other multi-modality models to better fuse environment feedback into task planning and execution [ 9 , 12 ] . However, recent works [ 6 , 16 , 17 ] have shown that LLMs may not always handle the whole process of converting natural language instructions into executable robot actions robustly without careful design of different components including symbol grounding, task decomposition, sub-task arrangement, formal logic translation, path generation, execution monitoring, and closed-loop feedback. Fig. 1 : Our method is able to interpret natural language instructions for a robot navigation task, translate them into an LTL formula free of syntax errors, and plan a robot path that satisfies the instructions. The figure shows an example of a Jackal robot fetching an umbrella, bringing it to a backpack, and finally navigating to a microwave oven. Many works have explored an end-to-end task planning approach where an LLM converts natural language instructions directly into robot actions [ 18 , 9 , 8 , 19 , 10 , 5 ] . In this paper, we consider an alternative modular task planning approach where an LLM converts natural language instructions into a task model first and then composes it with a robot model and uses task and motion planning to generate robot actions. To ground the concepts in natural language instructions, our approach relies on a semantic occupancy map [ 20 ] constructed from range observations and semantic segmentation observations [ 21 ] . The map captures the free and occupied space as well as the categories of objects around the robot, allowing a connection to the objects that the natural language instructions may be referring to. We introduce a novel code generation approach, LTLCodeGen, to convert the natural language instructions into an LTL formula. The LTL formula and the semantic occupancy map are provided to a motion planning algorithm to generate a collision-free path that satisfies the task requirements. An example of our task planning approach is shown in Fig. 1 . The contributions of this paper are summarized as follows. 1. We propose LTLCodeGen, a method to translate natural language to syntactically correct linear temporal logic formulas using code generation. 2. We develop a modular task planning approach that converts natural language instructions to LTL over semantic occupancy map and uses motion planning to generate robot paths that satisfy the task requirements. 3. We compare LTLCodeGen with NL2LTL [ 7 ] and LLMs fine-tuned for LTL translation [ 22 ] on both human-written and synthetic datasets. We also compare our task planning approach with end-to-end LLM-based task planning on the GameTraversalBenchmark (GTB) [ 6 ] . Additionally, we conduct real-world robot experiments with tasks of different difficulty levels, and perform ablation studies to analyze the importance of different components in LTLCodeGen. II Related work The components of natural language task planning include a) translating natural language instructions into a machine-interpretable format; b) constructing a scene representation with rich information to support instruction grounding and task planning; c) planning the task based on the translated instruction and the scene representation. II-A Natural Language Instruction Translation Regardless of whether LLMs understand human language, their ability to transform text input to different text outputs based on specific requirements is evident [ 23 ] . Since LLMs are trained on internet-scale text data, the common knowledge stored in the model is helpful for inferring the human intent embedded in the natural language instructions. Converting instructions into machine-interpretable format involves two challenges. One is to associate concepts, e.g. objects, in the instructions, to physical locations in space, which is known as symbol grounding. Whether LLMs solve symbol grounding well remains unclear [ 23 , 24 ] due to the debate on whether LLMs understand human language. The other challenge is to translate the symbol-grounded instruction to a machine-interpretable format that must correctly encode both the task objectives and constraints. Recent works use different task representation formats, including LTL [ 22 , 7 ] , STL [ 25 , 26 ] , PDDL [ 13 ] , code [ 8 , 10 , 15 , 11 ] , or others (e.g. waypoints [ 5 ] , action scores [ 4 ] , API calls [ 12 ] ). Because LLMs are usually trained with a large corpus of code examples, they excel in writing high-quality code that calls predefined APIs based on the instruction, as shown in CaP [ 8 ] , ProgPrompt [ 10 ] , and Demo2Code [ 15 ] . However, it is still a challenge to generate code that provides an optimal and semantically correct plan for direct task execution. Other works use LTL [ 7 ] or STL [ 25 ] , whose alignment with the user intent can be examined, to provide a task constraint for motion planning algorithms [ 27 , 28 , 29 ] . However, LLMs are not trained with various temporal logic examples and, even with carefully designed examples in the prompt, LLMs still generate syntactically incorrect LTL formulas occasionally. Some works [ 22 , 26 ] propose to fine-tune LLMs for translating temporal logic. However, it is still difficult to scale up as the LTL formula becomes more complicated. To obtain both the syntax robustness of code generation and the post-processing convenience of temporal logic, we develop LTLCodeGen, which uses LLMs to generate code that outputs LTL formulas. LTLCodeGen makes the whole system more robust to handle more complicated tasks and enables optimal planning for the instruction [ 29 , 7 ] . II-B Scene Representation A semantically annotated scene representation is essential for accurate instruction interpretation. It must support symbol grounding, collision checking, and task progress feedback to ensure safe and correct execution. Some works aim to build such representations, such as a semantic occupancy octree [ 20 ] , or a scene graph [ 30 , 31 , 32 , 33 , 34 ] . However, LLMs have a limited context window size, so the scene should be described to the LLM in a compact way. Existing works provide descriptions of involved objects only [ 8 , 25 , 11 ] or compress the scene in a concise format and then rely on the LLM to extract items related to the instruction [ 7 ] . Other works also consider obtaining scene information in different modalities [ 35 , 9 ] . II-C Task and Motion Planning Prior works focus on different aspects of task and motion planning for natural language instruction. For improved feasibility, SayCan [ 4 ] fuses robot-state-based action affordances into the LLM planning procedure. InnerMonologue [ 18 ] continually injects new sensor observations as feedback into the LLM planner to correct the execution online. Some works focus on multi-robot collaboration using a single LLM to reason about task decomposition and allocation [ 11 ] or a dialogue between multiple LLMs [ 5 ] . In contrast, our work focuses on generating an optimal path satisfying the LTL formula that correctly reflects the instruction. III Problem Statement Enabling mobile robots to navigate following human instructions in natural language is essential for effective human-robot interaction. We consider a mobile robot with state x_{t}\in{\cal X}\subseteq\mathbb{R}^{n} at time t equipped with an RGBD sensor. The robot first needs to construct a map of its environment, which stores information needed to understand the semantics of a human instruction. To ensure safe navigation, the map should also capture the presence of obstacles correctly. Definition 1 . A semantic occupancy map is a function m:{\cal X}\to{\cal C} , which associates a robot state \mathbf{x}\in{\cal X} to a semantic category m(\mathbf{x})\in{\cal C} present at state \mathbf{x} . The set {\cal C} is a set of semantic categories, including object classes, a free class that represents unoccupied free space, a null class that represents occupied space with unknown semantic category, and an unknown class for unobserved space. Fig. 2 shows an example of a semantic occupancy map. For example, given a semantic occupancy map of a house, the robot may be asked to tidy up the dining table and place the dishes in a dishwasher. Using the map, the robot can connect the instructions to the physical locations of the table and dishwasher and plan its motion to navigate to these locations safely and follow the sequence required by the instructions. To construct a connection between the natural language instructions and the semantic occupancy map, we define boolean propositions that capture the task requirements. Definition 2 . An atomic proposition is a boolean function p_{c}(\mathbf{x}) that evaluates true when the robot state \mathbf{x} is sufficiently close to a class c\in{\cal C} in a semantic occupancy map m : p_{c}(\mathbf{x})=\begin{cases}\text{true},&\exists\mathbf{y}\in{\cal X},m(\mathbf{y})=c,\left\|\mathbf{x}-\mathbf{y}\right\|\leq r_{c}\\ \text{false},&\text{otherwise},\end{cases} (1) where r_{c} is a distance threshold associated with class c . Denote the set of atomic propositions by {\cal A}{\cal P}=\{p_{c}|c\in{\cal C}\} . To capture which atomic propositions are satisfied at different robot states, we define a label map. Definition 3 . Given a semantic occupancy map m and atomic propositions {\cal A}{\cal P} , a label map l:{\cal X} \to 2^{{\cal A}{\cal P}} associates a robot state \mathbf{x} to the set of atomic propositions that evaluate true at \mathbf{x} , i.e., p_{c}(\mathbf{x})=\text{true} for all p_{c}\in l(\mathbf{x})\subseteq{\cal A}{\cal P} . The atomic propositions satisfied along a path \mathbf{x}_{1:T}:=\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{T} are obtained from the label map as l(\mathbf{x}_{1:T}):=l(\mathbf{x}_{1}),l(\mathbf{x}_{2}),\ldots l(\mathbf{x}_{T}) . The sequence l(\mathbf{x}_{1:T}) is called a word and it encodes the task requirements satisfied along the robot path \mathbf{x}_{1:T} . Finally, we use the notation l(\mathbf{x}_{1:T})\models\mu to indicate that all requirements of a task \mu are satisfied by a word l(\mathbf{x}_{1:T}) . We define the motion planning problem associated with a natural language task \mu as follows. Problem . Given a semantic occupancy map m , a natural language navigation task \mu defined in terms of semantic classes {\cal C} , a cost function d:{\cal X}\times{\cal X}\to\mathbb{R}_{>0} , and an initial robot state \mathbf{x}_{1}\in{\cal X} , plan a path \mathbf{x}_{1:T} that satisfies \mu with minimum cost: \displaystyle\min_{T\in\mathbb{N},\mathbf{x}_{1:T}} \displaystyle\sum_{t=1}^{T-1}d(\mathbf{x}_{t},\mathbf{x}_{t+1}) (2) \displaystyle\text{s.t.}\quad l(\mathbf{x}_{1:T})\models\mu; \displaystyle m(\mathbf{x}_{t})=\textsc{free},\;\;t=1,\dots,T. A key aspect of solving the problem above is to ground the task specification \mu to states in the map m with associated semantic classes and atomic propositions and to encode the dependencies among the atomic propositions according to \mu . In the next section, we propose an LLM code generation technique to generate LTL from \mu using the atomic propositions defined by the semantic occupancy map m . Fig. 2 : Semantic occupancy map of an office environment. Each voxel encodes both occupancy and semantic classification, represented by distinct colors: cyan indicates walls, orange and blue denote office tables, sky blue represents a cart, and lime green corresponds to carpeted floors. IV Robot Task Planning via LTLCodeGen Our approach to the task planning problem described in the previous section begins with constructing a semantic occupancy map, which we discuss in Sec. IV-A . Next, in Sec. IV-B , we introduce our main contribution, LTLCodeGen, a method to translate natural language to syntactically correct LTL formulas over semantic occupancy map atomic propositions using code generation. Finally, in Sec. IV-C , we present a task planning approach that takes the semantic occupancy map and the LTL formula and generates a collision-free robot path that satisfies the task requirements. Fig. 3 shows an overview. Fig. 3 : Overview of robot task planning using LTL code generation. Given a semantic occupancy map (left), we convert natural language instructions into a syntactically correct LTL formula using an LLM code generation approach (top). A label map of atomic propositions (bottom, middle), obtained from the map semantic information, and an Büchi automaton (top, right), obtained from the LTL formula, are used as inputs to a motion planning algorithm to generate a collision-free semantically valid robot path that executes the task. IV-A Semantic Occupancy Mapping Before the robot can interpret a desired task, it needs a representation of the environment that captures occupancy (for collision avoidance) and semantic classes (for task grounding). We assume that the robot is able to explore the environment first (either autonomously or through teleoperation) to construct a semantic occupancy map (Definition 1 ) using an RGB and range sensor. Our approach requires three components: an odometry algorithm (e.g., Direct LiDAR Odometry (DLO) [ 36 ] ) that estimates the robot’s pose, a semantic segmentation algorithm (e.g., YOLO [ 21 ] ) that assigns semantic labels from {\cal C} to image pixels in real time, and a semantic occupancy mapping algorithm (e.g., SSMI [ 20 ] ) to fuse semantically labeled point clouds, constructed from the segmentation and range measurements, into a semantic occupancy map. We chose DLO [ 36 ] for localization, YOLO [ 21 ] for semantic segmentation, and SSMI [ 20 ] for semantic occupancy mapping due to their efficiency, robustness, and the availability of open-source code. Other alternatives would also be compatible with our approach. While SSMI [ 20 ] constructs a 3D semantic occupancy map, in our experiments we consider a ground wheeled robot navigating on a 2D plane. We project the 3D map m_{\text{3D}} onto this plane and discard voxels outside of a vertical region of interest {\cal Z}=\{z|z_{\text{ground}}