Defining and Monitoring Complex Robot Activities via LLMs and Symbolic Reasoning

Francesco Argenziano, Elena Umili, Francesco Leotta, and Daniele Nardi · 2025

Read Defining and Monitoring Complex Robot Activities via LLMs and Symbolic Reasoning by Francesco Argenziano, Elena Umili, Francesco Leotta, and Daniele Nardi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16006.txt.

Introduction

Robots in farms and plants must do long jobs made of many small tasks, and the mix of tasks can shift on the fly. Francesco Argenziano and three co-authors set out an architecture that pairs large language models with symbolic planning. People state a high-level job in plain words, and the system turns that into steps a robot can follow.

Staff can also ask the robot what it has done, what it is doing, and what comes next. The team built the system with current tools and tested it in a real precision farming case. Measured results show the approach can track complex work in the field.

Catalog record

CREATORFrancesco Argenziano, Elena Umili, Francesco Leotta, and Daniele Nardi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-16006

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Defining and Monitoring Complex Robot Activities via LLMs and Symbolic Reasoning Francesco Argenziano, Elena Umili, Francesco Leotta, and Daniele Nardi Abstract Recent years have witnessed a growing interest in automating labor-intensive and complex activities, i.e., those consisting of multiple atomic tasks, by deploying robots in dynamic and unpredictable environments such as industrial and agricultural settings. A key characteristic of these contexts is that activities are not predefined: while they involve a limited set of possible tasks, their combinations may vary depending on the situation. Moreover, despite recent advances in robotics, the ability for humans to monitor the progress of high-level activities - in terms of past, present, and future actions - remains fundamental to ensure the correct execution of safety-critical processes. In this paper, we introduce a general architecture that integrates Large Language Models (LLMs) with automated planning, enabling humans to specify high-level activities (also referred to as processes) using natural language, and to monitor their execution by querying a robot. We also present an implementation of this architecture using state-of-the-art components and quantitatively evaluate the approach in a real-world precision agriculture scenario. I INTRODUCTION In recent years, there has been a significant increase in the interest and demand for automating complex and labor-intensive activities through the deployment of robotic systems. These activities, often encountered in industrial and agricultural domains, are typically composed of multiple, smaller atomic subtasksthat must be coordinated to achieve a larger goal. What makes these environments particularly challenging is their dynamic and unpredictable nature: the specific sequence and combination of tasks required can change based on real-time conditions, external events, or evolving objectives. Importantly, while the range of possible jobs is usually limited and known in advance, the structure and flow of the overall activity are not fixed and therefore must be adapted on the fly. In such contexts, human operators must maintain a clear understanding of ongoing high-level activities, both to ensure correctness and to make timely decisions. This includes being aware of what the system has done (past), what it is currently doing (present), and what it intends to do next (future). However, this kind of situational awareness is difficult to maintain in the absence of mechanisms for querying the system in a human-friendly way. High-level activities (in the following the term processes will be also used for brevity) can be described in different ways, usually employing formal methods and languages such as Petri Nets or temporal logics [ 1 ] . They can be imperative (i.e., only what is explicitly specified is allowed) or declarative (i.e., what is not forbidden is allowed). Their definition is usually transparent and independent from the actual functionalities provided by the participating actors. As a consequence, in order to actually enact them, they must be bound to a sequence of executable actions that can be obtained, for example, by using automated synthesis techniques (e.g., automated planning [ 2 ] ). In addition, when collaboration involves humans and robots, the way information is conveyed is fundamental from the point of view of usability, with speech being a natural way for a human to convey information with a robot [ 3 ] . Recently, the world of Natural Language Processing (NLP) has been revolutionized by the emergence of Large Language Models (LLMs), advanced computational models capable of operating on and generating sentences in natural language [ 4 ] . State-of-the-art large pre-trained language models (PLMs) such as BERT [ 5 ] , GPT [ 6 , 7 ] and LLaMA [ 8 ] are not domain-specific, and their output can be wrong or ambiguous (this phenomenon is referred to as hallucination ) when applied to specialized applications. Developing a domain-specific model requires a proper dataset and significant computing power for LLM training (or fine-tuning), which may not always be available. In this case, the in-context learning ability of LLMs paired with the prompt engineering approach, becomes useful, enabling zero-shot or few-shot learning without additional training data [ 9 , 10 ] . Users design specific prompt text, sometimes with few examples ( few-shot prompting ), to guide an LLM in generating desired responses. In this paper, we introduce a framework that enables users to specify high-level activities in natural language, making it accessible to non-experts. Additionally, it allows users to interact with the system during execution, asking questions about task progress, future steps, or explanations of behavior. This improves transparency, usability, and trust in robotic systems operating in open and evolving environments. In particular, we aim at the following design goals: • Not relying on a fixed set of processes. Conversely, the objective is to let humans define them on the fly, according to the needs encountered on the field, and to be aware of what the robot is going to do in order to satisfy the expressed goals; • Constraining processes to tasks actually executable by the robot on the environment. This means modeling the environment and possible actions formally; • Integrating process monitoring by human with other information related to the state of the world; • Relying on natural language alone for Human-Robot communication. We validate our approach with a full implementation of the proposed architecture using state-of-the-art LLMs and planning systems. The solution is evaluated in a real-world precision agriculture scenario, where it demonstrates the effectiveness and adaptability of the approach in supporting both autonomous task execution and intuitive human-robot interaction. The paper aims to show how integrating symbolic reasoning with modern LLMs can benefit users by (i) providing precise solutions using symbolic reasoning, (ii) simplifying interaction by shielding users from the complexity of symbolic reasoning formalisms, (iii) leveraging LLMs for natural language interaction, and (iv) relieve the LLM of performing planning tasks, which often leads to hallucinations. The paper is structured as follows. Section II introduces related works, contextualizing the work with respect to the state of the art. Section III describes the proposed approach from the conceptual and implementation point of view. Section IV evaluates the approach with a quantitative analysis in a challenging precision agriculture scenario. Section V concludes the paper with discussion and future works. II RELATED WORKS Behavioral modeling and process formalisms Early work modeled robot and human behaviors with Finite State Automata (FSA) [ 11 ] . FSAs are simple and intuitive but scale poorly: complex behaviors require exploding state spaces that are hard to design, maintain, and verify. To increase expressivity, researchers adopted richer formalisms such as BDI (Belief-Desire-Intention) architectures [ 12 ] , Petri nets [ 13 ] , and process-modeling techniques from business-process research [ 1 , 14 ] . BDI systems provide flexible, plan-selection semantics (intentions chosen from a plan library) that avoid committing designers to fixed action sequences, but they typically lack integrated automated planning and formal validation tooling. Petri nets and related imperative models reduce state explosion compared to FSAs but still require enumerating allowed behaviors a priori, which can be restrictive in open or human-interactive settings. Declarative process formalisms based on temporal logics (e.g., LTL) shift the specification burden from enumerating traces to stating constraints, enabling more compact specifications and supporting runtime verification; several robotics works exploit these properties for safety and constraint enforcement [ 15 , 16 ] . Our approach leverages the

Read the complete text