Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models

Casey Kennington \email [email protected], \name Pierre Lison \email [email protected], \name David Schlangen \email [email protected] · 2025

Read Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models by Casey Kennington \email [email protected], \name Pierre Lison \email [email protected], \name David Schlangen \email [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00953.txt.

Introduction

Casey Kennington, Pierre Lison, and David Schlangen present a 2025 review of word-by-word dialogue systems and what they teach the age of large language models. Today's models take in whole chunks and cannot revise mid-stream. Human talk, though, runs word by word, with repairs on the fly. The authors survey systems that listen, grasp, and speak in small steps, from sound intake to word choice.

The thinnest field, they find, is the step-by-step decision core that picks each next move. They list what a workable such core needs and what it means for robots with bodies. A robot that can be interrupted, corrected, and steered mid-sentence needs that core. Old lessons guard new builds from old mistakes.

Catalog record

CREATORCasey Kennington \email [email protected], \name Pierre Lison \email [email protected], \name David Schlangen \email [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00953

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models Casey Kennington \email [email protected], \name Pierre Lison \email [email protected], \name David Schlangen \email [email protected] Abstract Efforts towards endowing robots with the ability to speak have benefited from recent advancements in natural language processing, in particular large language models. However, current language models are not fully incremental, as their processing is inherently monotonic and thus lack the ability to revise their interpretations or output in light of newer observations. This monotonicity has important implications for the development of dialogue systems for human–robot interaction. In this paper, we review the literature on interactive systems that operate incrementally (i.e., at the word level or below it). We motivate the need for incremental systems, survey incremental modeling of important aspects of dialogue like speech recognition and language generation. Primary focus is on the part of the system that makes decisions, known as the dialogue manager. We find that there is very little research on incremental dialogue management, offer some requirements for practical incremental dialogue management, and the implications of incremental dialogue for embodied, robotic platforms in the age of large language models. Editor: Name Surname Submitted MM/YYYY; Accepted MM/YYYY; Published online MM/YYYY Keywords: spoken dialogue systems, incremental, human-robot interaction, dialogue management 1 Introduction Large Language Models (LLMs) have become more prominent in robotics, and for good reason. Williams et al. (2024) explain that LLMs can offer “quick-enabling of full-pipeline solutions” for many aspects of robots ranging from enabling robots to engage humans in spoken interaction to generating action plans ( Singh et al., 2023 ; Cohen et al., 2024 ; Singh et al., 2024 ; Mahadevan et al., 2024 ) and emotional behaviors ( Mishra et al., 2023 ) . While promising, some recent work has identified important qualities that LLMs lack which, if part of the model, would make interaction with robots seem more natural. A recent survey of spoken interaction on robots by Reimann et al. (2023) showcases a long history of research that spoken dialogue systems ( sds s) are key to endowing robots with handling common artifacts in spoken interaction which are not commonly found in text or written interaction such as turn-taking, requests for clarification, building common ground and mutual understanding. At the heart of their focus is the dialogue manager because both sds s and robots must make decisions about which actions they will take at any given moment, either by uttering a response, moving a robotic arm, or any other potential action within the capabilities of the robot. The dm (or corresponding robot action manager) not only decides which action to take, but also when to take that action; both are critical for natural interaction between robots and humans ( Lison and Kennington, 2023 ) . A recent review by Reimann et al. (2024) looked at dm in hri tasks and settings comparing how different systems divide the decision-making responsibilities, concluding that dm on robots is still rather a new field; more data, tasks, benchmarks, and discussions are needed. Fortunately, there exists a body of literature that spans over 25 years ( Allen et al., 2001 ) of success in developing and improving systems that enable humans to talk naturally to machines: incremental dialogue , meaning that processing happens at a fine-grained, word-by-word level. Comparisons between incremental and non-incremental systems have shown that incremental systems significantly improve system performance ( Ghigi et al., 2014a ) , are perceived by humans as being more natural ( Aist et al., 2007 ; Asri et al., 2014 ) and human-like and natural ( Edlund et al., 2008 ) , which suggests that the most appropriate systems for robots should be incremental, echoing the requirements of “robot-ready” sds for use in human-robot interaction ( hri ) settings ( Kennington et al., 2020 ) . LLM research can greatly benefit from this knowledge on incremental processing. Inoue et al. (2024b) points out that, for example, when humans engage in real-time dialogue with robotic agents, humans expect the robot to take seamless turns (i.e., without a gap in conversation, as happens within human-human dialogue) and they expect backchannels (e.g., nodding, or utterances like yeah , or uh huh ), neither of which are handled with common LLMs. Their work applied Voice Activity Projection (VAP) to enable LLMs to predict when a person might stop speaking so the LLM can respond at an appropriate time. Chiba and Higashinaka (2025) ’s recent VAP method also used an LLM to predict when to start speaking, and their work directly relied on incremental processing. According to the authors: “[…]even if systems are equipped with a natural turn-taking model, such a model will be ineffective if response generation cannot begin immediately once a turn-shift is detected. Incremental response generation is an approach that addresses this issue.” Using LLMs in incremental settings is a positive step, but more work is needed. Furthermore, other recent work Hudeček and Dusek (2023) asked if LLMs are all that is necessary for task-oriented dialogue (albeit outside of dialogue with robots) with some negatives (e.g., “LLMs underperform[…]” in important aspects of dialogue) and positives (e.g., “LLMs show the ability to guide the dialogue to a successful ending”). Finally, Wagner and Ultes (2024) investigated the usefulness of LLMs in dialogue interaction and found that they are effective, but need control and guidance to ensure that dialogue responses are coherent. Taken together, while LLMs can be employed for a wide range of dialogue processing tasks, they still suffer from a number of limitations when it comes to incrementality. In particular, while a LLM decoder can process any kind of input including word-level input, they are trained to act upon complete, sentence-level input, rendering them unable to produce behavior where input and output happen concurrently (an exception in recent full-duplex models shows potential ( Zhang et al., 2025 ) ). In contrast, humans must see and process individual words while reading text, and psycholinguistic research has shown that speech comprehension happens at a word or even sub-word level ( Tanenhaus and Spivey-Knowlton, 1995 ) . Moreover, another requirement of incremental processing is non-monotonicity ; i.e., that a model can react to change in input, for example when information coming from a speech recognizer is incorrect, the model needs to be able to revoke the erroneous input and change its internal state–causal language modeling is strictly monotonic, but incremental processing should allow for non-monotonic input. With chatbots, the text in, text out nature of the interaction is well-suited for LLMs, but the expectation of human-like conversation becomes more challenging when people interact with robots, because robotic interfaces have anthropomorphic characteristics. If, for example, a robot has what appear to be eyes, people expect that the robot can see them, or if the robot has an arm they expect the robot to be able to point or grasp objects. Furthermore, it has been shown that people antrhopomorphize robots for gender ( Reich-Stiebert and Eyssel, ; Eyssel and Hegel, 2012 ) , intelligence ( Novikova et al., 2015 ) , and even age ( Plane et al., 2018 ) depending on the robot’s morphology, size, and movements, which affects the expectations of how robots behave:

Read the complete text