Evaluating Efficiency and Engagement in Scripted and LLM-Enhanced Human-Robot Interactions This work was supported by the European Union’s Horizon 2020 research and innovation program under agreement 101017274 (DARKO). * Authors contributed equally to the work. Corresponding author tim.schreiter@tum
Tim Schreiter *, Jens V. Rüppel *, Rishi Hazra, Andrey Rudenko, Martin Magnusson, Achim J. Lilienthal · 2025
Read Evaluating Efficiency and Engagement in Scripted and LLM-Enhanced Human-Robot Interactions This work was supported by the European Union’s Horizon 2020 research and innovation program under agreement 101017274 (DARKO). * Authors contributed equally to the work. Corresponding author tim.schreiter@tum by Tim Schreiter *, Jens V. Rüppel *, Rishi Hazra, Andrey Rudenko, Martin Magnusson, Achim J. Lilienthal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12128.txt.
Introduction
Tim Schreiter and co-authors compare scripted and language-model-guided talks with a robot, in a 2025 study. Robot systems blend sensing, clear intent signals, and safe movement to work with people. When people act in odd ways, such systems can fail to adapt and recover. Large language models might add that flexibility, but better talk may not mean better results.
The team staged a factory robot task with approach, instruction, and object handling steps. One version ran on a fixed script, and the other used model-enhanced replies. Eye tracking and surveys measured speed, focus, and how people rated the robot. Users rated the smart version higher. On plain task measures, though, the scripted version matched it.
Catalog record
| CREATOR | Tim Schreiter *, Jens V. Rüppel *, Rishi Hazra, Andrey Rudenko, Martin Magnusson, Achim J. Lilienthal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12128
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Efficiency and Engagement in Scripted and LLM-Enhanced Human-Robot Interactions This work was supported by the European Union’s Horizon 2020 research and innovation program under agreement 101017274 (DARKO). * Authors contributed equally to the work. Corresponding author [email protected] Tim Schreiter *, Jens V. Rüppel *, Rishi Hazra, Andrey Rudenko, Martin Magnusson, Achim J. Lilienthal Abstract To achieve natural and intuitive interaction with people, HRI frameworks combine a wide array of methods for human perception, intention communication, human-aware navigation and collaborative action. In practice, when encountering unpredictable behavior of people or unexpected states of the environment, these frameworks may lack the ability to dynamically recognize such states, adapt and recover to resume the interaction. Large Language Models (LLMs), owing to their advanced reasoning capabilities and context retention, present a promising solution for enhancing robot adaptability. This potential, however, may not directly translate to improved interaction metrics. This paper considers a representative interaction with an industrial robot involving approach, instruction, and object manipulation, implemented in two conditions: (1) fully scripted and (2) including LLM-enhanced responses. We use gaze tracking and questionnaires to measure the participants’ task efficiency, engagement, and robot perception. The results indicate higher subjective ratings for the LLM condition, but objective metrics show that the scripted condition performs comparably, particularly in efficiency and focus during simple tasks. We also note that the scripted condition may have an edge over LLM-enhanced responses in terms of response latency and energy consumption, especially for trivial and repetitive interactions. I Introduction The overarching objective of Human-Robot Interaction (HRI) research is the development of robots capable of engaging with humans naturally and intuitively [ 1 , 2 ] . To reach this objective, researchers develop and combine various methods to perceive and communicate with people, navigate in human environments, and perform collaborative actions. In practice, such systems are typically implemented with partially scripted scenarios or pre-programmed behaviors [ 1 , 3 ] . While these approaches work well in controlled settings, they often lack the ability to adapt to inherently dynamic and unpredictable human behavior. Robots such as the NAO, commonly deployed as transmitters of social cues [ 4 , 5 ] by researchers, often utilize such scripted behaviors [ 6 ] . The lack of flexibility becomes particularly problematic when testing new behaviors in user studies, as minor deviations from the expected protocol by participants can lead to invalid experiment states or unsuccessful outcomes. Another method deployed to compensate for technical limitations is the Wizard of Oz method [ 7 ] . However, this approach does not only still suffer from limited contextual inputs and intervention capabilities, it also ultimately replaces human-robot interaction with human-human interaction, mediated by a robot [ 8 ] , always at the risk of failing to create the illusion of autonomous interaction. LLMs represent a promising opportunity to facilitate flexible and context-aware responses to user inputs, potentially adapting and recovering from unexpected interaction states. However, augmenting scripted interactions with LLM-enhanced actions is not trivial due to the complexity and cost of integrating these models into existing systems [ 9 ] . The integration process requires substantial computational infrastructure, specialized expertise in model deployment, and careful consideration of the system architecture. Additionally, researchers must address conversational coherence and manage latency in real-time interactions [ 10 ] . Moreover, the resource-intensive nature of LLMs, including their computational demands and energy consumption [ 11 ] , necessitates a careful cost-benefit analysis when considering an implementation in robotic systems. This paper compares a scripted human-robot interaction with one in which an LLM is used to generate responses dynamically for a collaborative pick-and-place task usually found in industrial applications [ 12 ] . We consider the participants’ visual attention as an input to the LLM framework and also use it to quantify their perception of the robot. Additionally, we measure the participants’ task performance and use questionnaires to assess their subjective ratings of the robot. By analyzing participants’ gaze patterns, we aim to understand if and how attention allocation differs between the two conditions, mainly focusing on key interaction moments and robot communication behaviors. Our experimental design employs a within-subjects approach, where participants experience both interaction conditions in random order, allowing for direct comparison of behavioral and perceptual differences. Our results offer valuable insights into the potential of LLMs to enhance classical HRI methods by making interactions more adaptive and robust against unexpected human behavior. We also emphasize the importance of selecting interaction approaches that match the specific demands and complexity of the task, taking into account factors such as technical implementation challenges, energy consumption, and the trade-off between adaptability and predictability in the user experience. Figure 1 : Setup for interacting with a NAO robot mounted on a forklift. The participant begins at the corridor entrance, the robot is located behind a wall. Interaction steps: Black: Carrying and placing a tin can; Red: Receiving instructions from the NAO robot; Green, Blue: Box delivery; Yellow: Interaction conclusion. Insets show egocentric eye-tracking with gaze sequences overlaid in brown . II Methods This paper considers a representative interaction with an industrial forklift robot mediated by an anthropomorphic NAO robot [ 5 ] . During the interaction, the robot asks the participant to approach, manipulate objects, and disengage. The interaction is implemented in two conditions: one where the steps are scripted and timed, and another where an LLM backbone is used to generate flexible actions, adapting to the interaction status. We aim to quantify the difference in robot perception and task execution between those two conditions. To that end we analyze gaze tracking and questionnaire responses, and measure energy consumption in both conditions by computing power across network nodes and API service calls. II-A Experiment Design Figure 2 : Sequence plan of the experiment. The robot is either controlled by an LLM (in blue ) or a pre-programmed schedule (PPS, in orange ). Roman numbers I–IV denote the key interaction steps, color coded according to Figure 1 . The study was conducted in a 10 m x 5 m corridor. Figure 1 illustrates a sample interaction including sample cutouts of the recorded gaze overlays to show what participants observed during the interaction. Participants began the experiment at one end of the corridor, holding a tin can with both hands. Their initial task (given by the experimenter) was to approach a table on their right and place the tin can there. During this stage, the robot they would later interact with was occluded and not visible to them. After the participant had placed the tin can on the table and approached the forklift ( \leq 3 m), the robot initiated interaction with a predefined phrase: “Hello, colleague. Please help me with my task.” The subsequent interaction followed a sequence plan implemented in two conditions (pre-programmed schedule, or PPS, and LLM-enhanced), as depicted in Figure 2 . The key difference between the two conditions was the interaction control mechanism. In the PPS condition, the robot’s behavior followed a fixed schedule based on the design in [ 5 ] , providing deterministic and predictable cues without adapting to the participant’s specific behavior. In contrast, LLM-enhanced responses utilized an interaction interface incorporating multiple visual and gaze attention cues and LLM reasoning